Software Testing Certification Study Guide 9789351194323, 9788177227116

Table of contents :
Cover
Title Page
Copyright
Contents at a Glance
Preface
Career Path for Test Engineers
About Software Testing Certification
Who Should Read this Book
Organization of the Book
Credits
Table of Contents
Chapter 1: Introduction
1.1 Power of Software
1.2 Challenges in Software Projects
1.3 Software Fiascos
1.3.1 Ariane 5
1.3.2 Patriot Missile
1.3.3 Mars Pathfinder
1.3.4 CT Scanner
1.3.5 The Great Bank Robbery
1.3.6 FBI Virtual Case File
1.4 Reasons for Software Failure
1.5 What is the Solution?
1.5.1 Software Quality Assurance
1.5.2 Software Testing
1.5.3 Code of Ethics
1.6 Software Testing Professionals
1.6.1 Skill Sets for Testing Professionals
1.6.2 Tasks Handled by Testing Professionals.
Summary
Review Questions
Chapter 2: Software Development Life Cycle and Quality Assurance
2.1 The Software Crisis
2.2 The Birth of Software Engineering
2.3 What is Software Engineering?
2.4 Why Software Engineering?
2.5 Is the Crisis Overcome?
2.6 Measuring Success of Software Project
2.6.1 Criteria for the Success of a Software Project
Quality
Time Factor
Project Budget
Maintaining Relationships
2.6.2 Client’s Expectations
2.7 Software Quality Assurance
2.7.1 The Software Process
2.7.2 Phases in Software Development Life Cycle
Market Analysis/Research
Feasibility Studies
Software Requirements Engineering
Software Design
Construction or Implementation
Testing
Maintenance
Retirement
2.8 Life Cycle Models
2.8.1 Waterfall Model
Requirements Specifications
System and Software Design
Implementation and Unit Testing
Integration and System Testing
Operation and Maintenance
Outputs of Different Stages in Waterfall Model
2.8.2 Prototyping Model
Advantages of Prototyping Model
2.8.3 Evolutionary Development Model
Advantages of Evolutionary Development Model
Disadvantages of Evolutionary Development Model
2.8.4 Spiral Model
Advantages of the Spiral Model
Disadvantages of the Spiral Model
2.8.5 Synchronize-and-Stabilize Model
Advantages of Synchronize-and-Stabilize Model
Disadvantages of Synchronize-and-Stabilize Model
2.8.6 Summary of Development Life Cycle Models
2.9 IEEE Standard for Developing Software Life Cycle Processes
2.10 Software Metrics
2.10.1 The Person Month
2.10.2 Product Metrics
2.10.3 Productivity Metrics
2.10.4 Quality Metrics
2.11 The Management Processes
2.12 Risk Management
2.12.1 Risk Item Identification
2.12.2 Risk Analysis
2.12.3 Risk Prioritization
2.12.4 Risk Control
2.13 Documentation
2.14 Quality Management Systems
2.14.1 Quality Standards
2.15 ISO 9000 Series Standards
Time Table for ISO 9000 Certification
2.16 Capability Maturity Model Integration for Software Engineering (CMMI-SW)
2.16.1 CMMI Framework
2.16.2 Levels of Software Process Maturity
2.16.3 Level 2 Process Areas
Requirements Management
Project Planning
Project Monitoring and Control
Supplier Agreement and Management
Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
2.16.4 Level 3 Process Areas
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Organizational Process Focus
Organizational Process Definition
Organizational Training
Integrated Project Management
Risk Management
Decision Analysis and Resolution
2.16.5 Level 4 Process Areas
Organizational Process Performance
Quantitative Project Management
2.16.6 Level 5 Process Areas
Organizational Innovation and Deployment
Causal Analysis and Resolution
2.16.7 Importance of CMMI
2.17 SPICE
Summary
Review Questions
Chapter 3: Fundamentals of Testing
3.1 What is Testing?
3.1.1 Testing versus Debugging
3.1.2 Testing and Bebugging
3.1.3 Verification and Validation
3.1.4 Root Cause Analysis
3.2 Significance of Testing
3.2.1 Cost of Quality
3.3 Psychology of Testing
3.4 Testing Choices
3.4.1 In-House Testing
3.4.2 Outsourcing
3.5 Who does the Testing?
3.5.1 Developers as Testers
3.5.2 Independent Testing Team
3.5.3 Buddy Testing
3.6 Testing Phases
3.6.1 V Model
3.6.2 Testing and Life Cycle Models
3.7 Testing the System
3.8 Testing Strategies
3.8.1 Static Testing
3.8.2 Dynamic Testing
3.8.3 Why Testing is Difficult?
Exhaustive testing
3.8.4 Test Case
Pesticide Paradox
Defect Clustering
3.8.5 Test Oracle
3.8.6 Test Software
3.8.7 Manual vs. Automated Testing
3.8.8 Testing Software of Different Technologies
3.9 Metrics in Testing Phase
3.10 When is Testing Complete?
3.10.1 Criteria for Completion of Testing
Defect Seeding
3.11 Risk-based Testing
3.11.1 Types of Risks
3.12 The Myths and Realities of Testing
Summary
Review Questions
Chapter 4: Testing Levels and Types
4.1 Testing Levels
4.1.1 Unit/Component Testing
4.1.2 Module Testing
4.1.3 Integration Testing
4.1.4 System Testing
4.1.5 Acceptance Testing
4.2 Testing Approaches
4.2.1 Static Testing vs. Dynamic Testing
4.2.2 Positive Testing vs. Negative Testing
4.2.3 Top-down Testing vs. Bottom-up Testing
4.2.4 Functional Testing vs. Structural Testing
Statement Coverage
Condition Coverage
Path Coverage
Function Coverage
Structural Testing at Integration Level
Structural Testing at System Level
Black Box Testing vs. White Box Testing at Code Level
4.2.5 Mutation Testing
4.2.6 Confirmation Testing
4.2.7 Regression Testing
4.3 Types of Testing
4.3.1 Smoke Testing
4.3.2 Black Box Testing
4.3.3 White Box Testing
4.3.4 Interface Testing
4.3.5 Use Case Testing
4.3.6 Gorilla Testing
4.3.7 Alpha Testing
4.3.8 Beta Testing
4.3.9 Field Trial/Operational Testing
4.3.10 Performance Testing/Load Testing
4.3.11 Stress Testing
4.3.12 Accessibility Testing
4.3.13 Conformance Testing
4.3.14 Internationalization Testing
4.3.15 Security Testing
4.3.16 Maintenance Testing
4.3.17 Acceptance Testing
4.3.18 Documentation Testing
Summary
Review Questions
Chapter 5: Static Testing Techniques
5.1 Static Testing
5.1.1 Advantages of Static Testing
5.2 Manual Reviews
5.2.1 Formal Review Process
Define Review Objectives
Identify the Review Team
Plan the Meeting
5.2.2 Informal Reviews
5.2.3 Walkthroughs
5.2.4 Inspections
5.2.5 Making Reviews Successful
5.2.6 Checklists
SRS Checklist
Design Checklist
GUI Checklist
Web Site Checklist
5.3 Formal Code Reviews
5.3.1 Coding Guidelines
5.3.2 Programming Style
5.3.3 C Coding Guidelines
5.3.4 Code Optimization
5.3.5 Java Coding Guidelines
Backup Policy and Schedules during Development Phase
Project Folder
Java Coding Conventions
Naming Methods
Naming Getters
Naming Setters
Naming Constructors
Naming Fields (Fields/Properties)
Naming Components/Widgets
Naming Constants
Naming Local Variables
Naming Streams
Naming Loop Counters
Naming Exception Objects
Naming Parameters (Arguments)
Naming Classes, interfaces and packages
What to Document
5.4 Static Analysis using Tools
5.4.1 Tool for Readability Improvement/ Indenting
5.4.2 Portability Testing Tool
5.4.3 Symbolic Execution
Summary
Review Questions
Chapter 6: Dynamic Testing and Test Case Design Techniques
6.1 Dynamic Testing
6.2 Review Work Products
6.3 Identify Test Objectives
6.4 Test Specifications and Test Design
6.5 Design Test Cases
6.5.1 Black Box Test Case Design Techniques
Specifications Based Testing
Equivalence Partitioning
Boundary Value Analysis
Decision Tables
Cause Effect Analysis
State Transition Diagrams
Use Case Scenarios
6.5.2 White Box Test Case Design Techniques
Code Coverage
Internal Boundary Value Testing
Structural Testing at Module Level
Structural Testing at System Level
6.5.3 Experience-based Test Case Design Techniques
Error Guessing
Exploratory Testing
6.5.4 Case Study #1: Test Cases for an IVR System
6.5.5 Case Study #2: Test Cases for Fingerprint Recognition System
6.6 Document Test Cases
6.7 Execute Test Cases
6.8 Generate Incident Report/Anomaly Report
6.9 Log the Defects
6.10 Test Documentation Standards
6.11 Formal Methods of Testing
Summary
Review Questions
Detail Questions
Chapter 7: Managing the Testing Process
7.1 Management Commitment
7.1.1 Organization Structure
7.2 Testing Process Management
7.2.1 Options for Managers
COTS Software
Software Development
7.2.2 Testing Process Management Activities
7.3 Planning, Budgeting and Scheduling the Testing Phase
7.3.1 Test Plan
7.4 Alignment of the Process to the Project
7.5 Team Formation
7.6 Infrastructure
7.6.1 Testing Tools
7.7 Reviewing, Monitoring and Risk Management
7.7.1 Risk Management
7.7.2 Test Reports
7.8 Metrics
7.8.1 Software Reliability
7.9 Defect Tracking
7.9.1 Classification of Defects
7.10 Configuration Management
7.11 Test Closure and Process Improvement
7.11.1 Software Testing Maturity Model (SW-TMM)
7.12 Information Security
Summary
Review Questions
Chapter 8: Software Testing Tools
8.1 Need for Tools
8.2 Classification of Tools
8.2.1 Functional/Regression Testing Tools
8.2.2 Performance/Load Testing Tools
8.2.3 Testing Process Management Tools
8.3 Benefits of Tools
8.4 Risks Associated with the Tools
8.5 Does your Organization Need Tools?
8.6 Selecting Tools
8.7 Introducing the Tools in the Testing Process
8.8 WinRunner
8.8.1 Testing an Application by using WinRunner
Recording Test Cases
Saving the GUI Map File
Loading the GUI Map File
Running a Test Case
8.8.2 Testing the Standard Calculator Application
Testing the Functionality of Application
Calling the Test Cases using call Function
8.8.3 Test Script Language (TSL)
8.9 LoadRunner
8.9.1 Creating Vuser Script by using Virtual User Generator
8.9.2 Creating Virtual Users by using LoadRunner Controller
8.10 TestDirector
8.10.1 Testing Management Process
Test Requirements Management
Test Planning
Test Execution
Test Results Analysis
8.10.2 Managing the Testing Process using TestDirector
Creating a New Project
Creating the Test Plan
Running the Automated Test
Tracking Defects
Summary
Review Questions
Chapter 9: Code of Ethics for Software Professionals
9.1 Human Ethics
9.2 Professional Ethics
9.3 Ethical Issues in Software Engineering
9.4 Code of Ethics for Software Professionals
9.4.1 Software Engineering Code of Ethics and Professional Practice
Short Version: Preamble
Full Version: Preamble
Principles
9.5 Ethical Issues: Right versus Wrong
Summary
Review Questions
Appendix A: Acronyms and Abbreviations
Appendix B: Glossary
Appendix C: Internet Resources and References
Internet Resources
References
Appendix D: Answers to Review Questions
Appendix E: Frequently Asked Questions
Index

Citation preview

Index

388

©copyright 2006 by Dreamtech Press, 19-A, Ansari Road, Daryaganj, New Delhi-110002.

This book may not be duplicated in any way without the express written consent of the publisher, except in the form of brief excerpts or quotations for the purposes of review. The information contained herein is for the personal use of the reader and may not be incorporated in any commercial programs, other books, databases, or any kind of software without written consent of the publisher. Making copies of this book or any portion for any purpose other than your own is a violation of copyright laws.

Limits of Liability/disclaimer of Warranty : The author and publisher have used their best efforts in preparing this book. The author make no representation or warranties with respect to the accuracy or completeness of the contents of this book, and specifically disclaim any implied warranties of merchantability or fitness of any particular purpose. There are no warranties which extend beyond the descriptions contained in this paragraph. No warranty may be created or extended by sales representatives or written sales materials. The accuracy and completeness of the information provided herein and the opinions stated herein are not guaranteed or warranted to produce any particulars results, and the advice and strategies contained herein may not be suitable for every individual. Neither Dreamtech Press nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Trademarks : All brand names and product names used in this book are trademarks, registered trademarks, or trade names of their respective holders. Dreamtech Press is not associated with any product or vendor mentioned in this book.

ISBN: 978-81-7722-711-6

ISBN: 978-93-5004-679-1 (ebk)

Contents at a Glance Chapter 1: Introduction .................................................................................................................. 1 Chapter 2: Software Development Life Cycle and Quality Assurance ..................................... 17 Chapter 3: Fundamentals of Testing........................................................................................... 89 Chapter 4: Testing Levels and Types........................................................................................ 127 Chapter 5: Static Testing Techniques....................................................................................... 165 Chapter 6: Dynamic Testing and Test Case Design Techniques ........................................... 193 Chapter 7: Managing the Testing Process ............................................................................... 219 Chapter 8: Software Testing Tools............................................................................................ 255 Chapter 9: Code of Ethics for Software Professionals............................................................ 343 Appendix A: Acronyms and Abbreviations .............................................................................. 357 Appendix B: Glossary ................................................................................................................ 363 Appendix C: Internet Resources and References.................................................................... 375 Appendix D: Answers to Review Questions............................................................................. 379 Appendix E: Frequently Asked Questions ............................................................................... 383 Index ............................................................................................................................................ 387

Index

388

Preface In the last one decade, software testing has become a separate industry. With outsourcing of software testing activities by many major software development and IT user organizations, the demand for software testing/quality assurance professionals is growing exponentially and their salaries are skyrocketing. However, as testing is a very challenging task and the industry requires high-quality testing professionals, in recent years, certification as testing engineers is becoming a must for all budding professionals who want to have a career in this field. To become a good testing professional, a sound theoretical knowledge about testing, and a strong practical knowledge about using the tools are a must. Certification examinations focus on the understanding of the important concepts of software testing and software quality assurance without any specific orientation towards one particular tool. This book aims at providing the necessary knowledge in understanding the concepts of software testing and software quality assurance so that you can take any internationally recognized software testing/quality assurance certification examination and come out with flying colors. Also, equipped with this knowledge, you can do a great job as a testing and quality assurance professional in your career; and contribute in developing reliable software for different applications, which in turn improves the quality of life of everyone on this earth.

Career Path for Test Engineers It is a misconception among the young graduates that software testing provides only jobs on a short-term basis and in the long run, there is no guarantee for their careers. Please understand that this is a very wrong notion. Software testing continues to be a lucrative career as long as software development has to be done! Testing engineers are the most qualified people to emerge as quality managers in future. Quality is the ‘mantra’ of every software company, and nowadays, quality managers are the most important people in every software company. If testing engineers continue to keep in touch with software development tools and programming, then they will be the best people for taking up software development as they can develop high-quality–reliable software. It would be really good if every CEO of software organizations insists that the prerequisite for taking up software development professional is software testing—at least for one year. In such a case, we will never have to use a software package that has bugs.

Preface

vi

About Software Testing Certification The demand for test engineers is growing exponentially. The number of people aspiring for a test engineer’s job is also growing rapidly. When the competition is very high, it becomes difficult for software companies to select a few candidates from a lot of applicants. So, the companies resort to short-listing the applications using some criteria— the criterion that is most common is certification as software testing professional. Candidates with certification from national/international bodies do have an edge over others in a tight job market. Software engineering or software testing is taught as a subject at undergraduate level. But then, the syllabus of each university is different, the examination patterns are different, and hence when software development organizations have to recruit testing professionals, it will be difficult to assess their knowledge. Hence, certification from an internationally recognized organization is very important for individuals. Many software development organizations are also keen on recruiting testing engineers who have obtained the certification because their evaluation is simplified. Hence, certification is useful for individuals because it increases their market value, visibility, and professionalism. It is also of great benefit to the organization for the same reasons. And that is why, nowadays, all individuals are taking the certification exams and the organizations are insisting on recruiting individuals with certification. Also, the client who gives testing projects will have confidence on the organization’s abilities because of the presence of certified test engineers, and not doubt in the testing capabilities of the manpower of the organization. International Software Testing Qualification Board (ISTQB) certification is one of the most widely recognized certifications. Their certification examinations are conducted in many countries through the local boards. Indian Testing Board (ITB) is the Indian arm of ISTQB which conducts the certification examination in India. “ISTQB Certified Tester” examination is at two levels—foundation level and advanced level. Both examinations are of multiple-choice questions. Persons with little experience in testing, but with sound knowledge can take the foundation level examination. Advanced level examination is aimed at experienced people with sound practical knowledge. British Computer Society’s Information Systems Education Board (ISEB) also conducts certification examination and this certification leads to ISTQB certified testing qualification. This certification is also conducted at two levels—foundation certification and practitioner certification. Quality Assurance International (QAI) also conducts certification examinations for test engineers and quality assurance professionals. QAI’s Certified Software Tester (CSTE) and Certified Software Quality Analyst (CSQA) examinations are of 4½ hour duration. The examinations have multiple-choice questions, essays, and short answer questions. In all the examinations, the candidate’s knowledge on the fundamental concepts of software testing and quality assurance is tested. CSTE and CSQA certification examinations are for working professionals—not for fresh graduates. The syllabus for these examinations is

Preface

vii based on the Common Body of Knowledge (CBOK) framed by QAI for these two examinations.

Who Should Read this Book This book is addressed to fresh graduates and those working in software development/testing, who would like to obtain certification as software testing professionals. Even if you are not interested in certification, this book will be of immense use in understanding the important concepts of software testing and quality assurance. Many universities are now introducing software testing as a separate subject for computer science/information technology students. This book can serve as a textbook for this subject. Software project managers and quality analysts can also get an insight into the intricacies of software testing by reading this book.

Organization of the Book This book is organized into nine chapters. Chapter 1 gives an introduction to the field of software development and software testing. The importance of developing quality software and how software testing helps in achieving this goal are discussed. Chapter 2 gives an overview of software development life cycle and quality assurance. Various phases of software development are discussed and the different life cycle models used for different types of projects are given. An overview of software quality assurance is presented and the important features of ISO 9001:2000 and Capability Maturity Model Integration (CMMI) are described. Chapter 3 lays the foundation of software testing by explaining the important concepts of testing and the testing process. Chapter 4 deals with testing levels and testing methods. The testing to be carried out at different levels or phases of development is explained. The various types of tests to be carried out on the software are also explained with examples. Chapter 5 is about static testing—how to carry out testing without executing the code. Static testing is very important to develop quality software and the different methods used for static testing are explained. Chapter 6 is about dynamic testing and test case design techniques. How to do the testing after executing the program and how to design test cases are discussed in detail with illustrative examples. To design test cases that will bring out the defects in the software, with the aim of reducing the testing effort and testing time, is the most

Preface

viii challenging task in software testing and this aspect is covered in this chapter with examples and case studies. Chapter 7 discusses the details of managing the testing process. How to estimate the effort required for testing, how to do a meticulous planning of the testing process, how to monitor and review the testing process, etc. are discussed in detail. Chapter 8 gives an overview of software testing tools. The need for testing tools, different types of testing tools, and how to select a tool are discussed. An overview of WinRunner, a functional testing tool; LoadRunner, a performance testing tool; and TestDirector, a testing process management tool is also given in this chapter. Chapter 9 discusses the ethical issues in software development. The code of ethics and professional practice, formulated by IEEE Computer Society and ACM is presented in this chapter. Note that a basic understanding of this code of ethics is a must to obtain certification as software testing professional/quality assurance professional from international bodies. Appendix A gives a list of acronyms and abbreviations. Appendix B explains the important terms used in software quality assurance and testing literature. Appendix C lists the important reference books and Internet resources from which you can get a wealth of information on software quality assurance and software testing. Appendix D gives answers to the review questions given at the end of each chapter. Appendix E lists frequently asked questions in job interviews for test engineers and quality assurance specialists.

Credits The syllabi formulated by ISTQB, Information Systems Education Board (ISEB) U.K., and Quality Assurance International (QAI)—for their certification examinations—are the basis for the contents of this book. All the figures are the handiwork of Mr. Vipul Pawaskar.

Table of Contents Chapter 1: Introduction .................................................................................................................. 1 1.1 Power of Software .................................................................................................................................... 1 1.2 Challenges in Software Projects ............................................................................................................. 2 1.3 Software Fiascos ....................................................................................................................................... 2 1.3.1 Ariane 5 ..........................................................................................................................................................3 1.3.2 Patriot Missile................................................................................................................................................4 1.3.3 Mars Pathfinder.............................................................................................................................................5 1.3.4 CT Scanner .....................................................................................................................................................6 1.3.5 The Great Bank Robbery..............................................................................................................................6 1.3.6 FBI Virtual Case File .....................................................................................................................................7

1.4 Reasons for Software Failure .................................................................................................................. 8 1.5 What is the Solution? ............................................................................................................................... 8 1.5.1 Software Quality Assurance........................................................................................................................9 1.5.2 Software Testing..........................................................................................................................................10 1.5.3 Code of Ethics..............................................................................................................................................10

1.6 Software Testing Professionals............................................................................................................. 11 1.6.1 Skill Sets for Testing Professionals...........................................................................................................12 1.6.2 Tasks Handled by Testing Professionals. ...............................................................................................13

Summary ....................................................................................................................................................... 14 Review Questions ........................................................................................................................................ 15 Chapter 2: Software Development Life Cycle and Quality Assurance ..................................... 17 2.1 The Software Crisis ................................................................................................................................ 17 2.2 The Birth of Software Engineering....................................................................................................... 18 2.3 What is Software Engineering? ............................................................................................................ 18 2.4 Why Software Engineering? ................................................................................................................. 19 2.5 Is the Crisis Overcome? ......................................................................................................................... 21 2.6 Measuring Success of Software Project ............................................................................................... 21 2.6.1 Criteria for the Success of a Software Project .........................................................................................22 2.6.2 Client’s Expectations ..................................................................................................................................28

2.7 Software Quality Assurance ................................................................................................................. 29 2.7.1 The Software Process..................................................................................................................................29 2.7.2 Phases in Software Development Life Cycle..........................................................................................30

2.8 Life Cycle Models................................................................................................................................... 39 2.8.1 Waterfall Model ..........................................................................................................................................39

Table of Contents

x 2.8.2 Prototyping Model......................................................................................................................................42 2.8.3 Evolutionary Development Model ..........................................................................................................44 2.8.4 Spiral Model.................................................................................................................................................45 2.8.5 Synchronize-and-Stabilize Model ............................................................................................................47 2.8.6 Summary of Development Life Cycle Models .......................................................................................50

2.9 IEEE Standard for Developing Software Life Cycle Processes......................................................... 51 2.10 Software Metrics................................................................................................................................... 52 2.10.1 The Person Month.....................................................................................................................................52 2.10.2 Product Metrics .........................................................................................................................................52 2.10.3 Productivity Metrics.................................................................................................................................53 2.10.4 Quality Metrics..........................................................................................................................................53

2.11 The Management Processes ................................................................................................................ 54 2.12 Risk Management................................................................................................................................. 54 2.12.1 Risk Item Identification............................................................................................................................55 2.12.2 Risk Analysis .............................................................................................................................................57 2.12.3 Risk Prioritization .....................................................................................................................................57 2.12.4 Risk Control ...............................................................................................................................................57

2.13 Documentation ..................................................................................................................................... 58 2.14 Quality Management Systems............................................................................................................ 59 2.14.1 Quality Standards .....................................................................................................................................59

2.15 ISO 9000 Series Standards ................................................................................................................... 60 2.16 Capability Maturity Model Integration for Software Engineering (CMMI-SW) ......................... 69 2.16.1 CMMI Framework ....................................................................................................................................70 2.16.2 Levels of Software Process Maturity .....................................................................................................70 2.16.3 Level 2 Process Areas ...............................................................................................................................73 2.16.4 Level 3 Process Areas ...............................................................................................................................74 2.16.5 Level 4 Process Areas ...............................................................................................................................76 2.16.6 Level 5 Process Areas ...............................................................................................................................77 2.16.7 Importance of CMMI................................................................................................................................78

2.17 SPICE ..................................................................................................................................................... 78 Summary ....................................................................................................................................................... 81 Review Questions ........................................................................................................................................ 83 Chapter 3: Fundamentals of Testing........................................................................................... 89 3.1 What is Testing? ..................................................................................................................................... 89 3.1.1 Testing versus Debugging.........................................................................................................................90 3.1.2 Testing and Bebugging ..............................................................................................................................90 3.1.3 Verification and Validation .......................................................................................................................91 3.1.4 Root Cause Analysis...................................................................................................................................91

3.2 Significance of Testing........................................................................................................................... 92 3.2.1 Cost of Quality ............................................................................................................................................93

Table of Contents

xi 3.3 Psychology of Testing............................................................................................................................ 94 3.4 Testing Choices....................................................................................................................................... 96 3.4.1 In-House Testing.........................................................................................................................................97 3.4.2 Outsourcing .................................................................................................................................................98

3.5 Who does the Testing?........................................................................................................................... 98 3.5.1 Developers as Testers .................................................................................................................................99 3.5.2 Independent Testing Team......................................................................................................................100 3.5.3 Buddy Testing ...........................................................................................................................................100

3.6 Testing Phases ...................................................................................................................................... 101 3.6.1 V Model ......................................................................................................................................................101 3.6.2 Testing and Life Cycle Models ...............................................................................................................103

3.7 Testing the System ............................................................................................................................... 104 3.8 Testing Strategies ................................................................................................................................. 105 3.8.1 Static Testing..............................................................................................................................................105 3.8.2 Dynamic Testing .......................................................................................................................................106 3.8.3 Why Testing is Difficult? .........................................................................................................................106 3.8.4 Test Case.....................................................................................................................................................110 3.8.5 Test Oracle..................................................................................................................................................112 3.8.6 Test Software .............................................................................................................................................113 3.8.7 Manual vs. Automated Testing ..............................................................................................................114 3.8.8 Testing Software of Different Technologies .........................................................................................114

3.9 Metrics in Testing Phase...................................................................................................................... 116 3.10 When is Testing Complete? .............................................................................................................. 116 3.10.1 Criteria for Completion of Testing.......................................................................................................117

3.11 Risk-based Testing ............................................................................................................................. 118 3.11.1 Types of Risks..........................................................................................................................................119

3.12 The Myths and Realities of Testing.................................................................................................. 120 Summary ..................................................................................................................................................... 122 Review Questions ...................................................................................................................................... 123 Chapter 4: Testing Levels and Types........................................................................................ 127 4.1 Testing Levels ....................................................................................................................................... 127 4.1.1 Unit/Component Testing........................................................................................................................128 4.1.2 Module Testing .........................................................................................................................................129 4.1.3 Integration Testing....................................................................................................................................129 4.1.4 System Testing...........................................................................................................................................131 4.1.5 Acceptance Testing ...................................................................................................................................131

4.2 Testing Approaches ............................................................................................................................. 134 4.2.1 Static Testing vs. Dynamic Testing ........................................................................................................134 4.2.2 Positive Testing vs. Negative Testing....................................................................................................134 4.2.3 Top-down Testing vs. Bottom-up Testing ............................................................................................135

Table of Contents

xii 4.2.4 Functional Testing vs. Structural Testing .............................................................................................135 4.2.5 Mutation Testing.......................................................................................................................................140 4.2.6 Confirmation Testing ...............................................................................................................................141 4.2.7 Regression Testing ....................................................................................................................................141

4.3 Types of Testing ................................................................................................................................... 142 4.3.1 Smoke Testing ...........................................................................................................................................142 4.3.2 Black Box Testing ......................................................................................................................................143 4.3.3 White Box Testing.....................................................................................................................................144 4.3.4 Interface Testing........................................................................................................................................145 4.3.5 Use Case Testing .......................................................................................................................................147 4.3.6 Gorilla Testing ...........................................................................................................................................148 4.3.7 Alpha Testing ............................................................................................................................................149 4.3.8 Beta Testing................................................................................................................................................149 4.3.9 Field Trial/Operational Testing .............................................................................................................150 4.3.11 Stress Testing ...........................................................................................................................................153 4.3.12 Accessibility Testing...............................................................................................................................154 4.3.13 Conformance Testing .............................................................................................................................155 4.3.14 Internationalization Testing ..................................................................................................................156 4.3.15 Security Testing.......................................................................................................................................157 4.3.16 Maintenance Testing ..............................................................................................................................158 4.3.17 Acceptance Testing .................................................................................................................................158 4.3.18 Documentation Testing..........................................................................................................................160

Summary ..................................................................................................................................................... 161 Review Questions ...................................................................................................................................... 162 Chapter 5: Static Testing Techniques....................................................................................... 165 5.1 Static Testing......................................................................................................................................... 165 5.1.1 Advantages of Static Testing...................................................................................................................166

5.2 Manual Reviews ................................................................................................................................... 168 5.2.1 Formal Review Process ............................................................................................................................168 5.2.2 Informal Reviews ......................................................................................................................................170 5.2.3 Walkthroughs ............................................................................................................................................170 5.2.4 Inspections .................................................................................................................................................170 5.2.5 Making Reviews Successful ....................................................................................................................171 5.2.6 Checklists ...................................................................................................................................................172

5.3 Formal Code Reviews.......................................................................................................................... 174 5.3.1 Coding Guidelines ....................................................................................................................................175 5.3.2 Programming Style...................................................................................................................................175 5.3.3 C Coding Guidelines ................................................................................................................................176 5.3.4 Code Optimization ...................................................................................................................................178 5.3.5 Java Coding Guidelines ...........................................................................................................................179

Table of Contents

xiii 5.4 Static Analysis using Tools ................................................................................................................. 185 5.4.1 Tool for Readability Improvement/ Indenting ...................................................................................186 5.4.2 Portability Testing Tool ...........................................................................................................................186 5.4.3 Symbolic Execution ..................................................................................................................................189

Summary ..................................................................................................................................................... 189 Review Questions ...................................................................................................................................... 190 Chapter 6: Dynamic Testing and Test Case Design Techniques ........................................... 193 6.1 Dynamic Testing .................................................................................................................................. 193 6.2 Review Work Products........................................................................................................................ 194 6.3 Identify Test Objectives ....................................................................................................................... 194 6.4 Test Specifications and Test Design................................................................................................... 195 6.5 Design Test Cases................................................................................................................................. 196 6.5.1 Black Box Test Case Design Techniques ...............................................................................................197 6.5.2 White Box Test Case Design Techniques ..............................................................................................206 6.5.3 Experience-based Test Case Design Techniques .................................................................................209 6.5.4 Case Study #1: Test Cases for an IVR System ......................................................................................209 6.5.5 Case Study #2: Test Cases for Fingerprint Recognition System .......................................................211

6.6 Document Test Cases........................................................................................................................... 211 6.7 Execute Test Cases ............................................................................................................................... 212 6.8 Generate Incident Report/Anomaly Report..................................................................................... 213 6.9 Log the Defects ..................................................................................................................................... 214 6.10 Test Documentation Standards ........................................................................................................ 214 6.11 Formal Methods of Testing ............................................................................................................... 215 Summary ..................................................................................................................................................... 216 Review Questions ...................................................................................................................................... 216 Detail Questions .................................................................................................................................................217

Chapter 7: Managing the Testing Process ............................................................................... 219 7.1 Management Commitment................................................................................................................. 219 7.1.1 Organization Structure ............................................................................................................................221

7.2 Testing Process Management ............................................................................................................. 222 7.2.1 Options for Managers ..............................................................................................................................223 7.2.2 Testing Process Management Activities ...............................................................................................225

7.3 Planning, Budgeting and Scheduling the Testing Phase ................................................................ 226 7.3.1 Test Plan .....................................................................................................................................................229

7.4 Alignment of the Process to the Project............................................................................................. 230 7.5 Team Formation ................................................................................................................................... 231 7.6 Infrastructure ........................................................................................................................................ 234 7.6.1 Testing Tools..............................................................................................................................................234

Table of Contents

xiv 7.7 Reviewing, Monitoring and Risk Management ............................................................................... 235 7.7.1 Risk Management .....................................................................................................................................237 7.7.2 Test Reports ...............................................................................................................................................237

7.8 Metrics ................................................................................................................................................... 238 7.8.1 Software Reliability ..................................................................................................................................239

7.9 Defect Tracking..................................................................................................................................... 240 7.9.1 Classification of Defects ...........................................................................................................................241

7.10 Configuration Management.............................................................................................................. 242 7.11 Test Closure and Process Improvement.......................................................................................... 245 7.11.1 Software Testing Maturity Model (SW-TMM)...................................................................................246

7.12 Information Security .......................................................................................................................... 247 Summary ..................................................................................................................................................... 251 Review Questions ...................................................................................................................................... 252 Chapter 8: Software Testing Tools............................................................................................ 255 8.1 Need for Tools ...................................................................................................................................... 255 8.2 Classification of Tools.......................................................................................................................... 256 8.2.1 Functional/Regression Testing Tools....................................................................................................259 8.2.2 Performance/Load Testing Tools ..........................................................................................................262 8.2.3 Testing Process Management Tools.......................................................................................................265

8.3 Benefits of Tools ................................................................................................................................... 266 8.4 Risks Associated with the Tools......................................................................................................... 267 8.5 Does your Organization Need Tools? ............................................................................................... 267 8.6 Selecting Tools ...................................................................................................................................... 268 8.7 Introducing the Tools in the Testing Process.................................................................................... 270 8.8 WinRunner............................................................................................................................................ 271 8.8.1 Testing an Application by using WinRunner.......................................................................................272 8.8.2 Testing the Standard Calculator Application.......................................................................................276 8.8.3 Test Script Language (TSL) .....................................................................................................................284

8.9 LoadRunner .......................................................................................................................................... 286 8.9.1 Creating Vuser Script by using Virtual User Generator.....................................................................288 8.9.2 Creating Virtual Users by using LoadRunner Controller ..................................................................293

8.10 TestDirector......................................................................................................................................... 301 8.10.1 Testing Management Process ...............................................................................................................302 8.10.2 Managing the Testing Process using TestDirector ............................................................................304

Summary ..................................................................................................................................................... 338 Review Questions ...................................................................................................................................... 340 Chapter 9: Code of Ethics for Software Professionals............................................................ 343 9.1 Human Ethics ....................................................................................................................................... 343

Table of Contents

xv 9.2 Professional Ethics ............................................................................................................................... 344 9.3 Ethical Issues in Software Engineering ............................................................................................. 345 9.4 Code of Ethics for Software Professionals ........................................................................................ 346 9.4.1 Software Engineering Code of Ethics and Professional Practice ......................................................347

9.5 Ethical Issues: Right versus Wrong.................................................................................................... 354 Summary ..................................................................................................................................................... 355 Review Questions ...................................................................................................................................... 355 Appendix A: Acronyms and Abbreviations .............................................................................. 357 Appendix B: Glossary ................................................................................................................ 363 Appendix C: Internet Resources and References.................................................................... 375 Appendix D: Answers to Review Questions............................................................................. 379 Appendix E: Frequently Asked Questions ............................................................................... 383 Index ............................................................................................................................................ 387

Index

388

Chapter

Introduction Chapter Objectives y y y y y y y

1

The importance of software The challenges in software development The software fiascos Reasons for software failures How to avoid software failures? Testing Professional’s role Skills required for testing professionals

1.1 Power of Software It is not an exaggeration if we say that, the world will come to a standstill if all the computers on the earth stop working. Nowadays, computers are used in every walk of life— education, entertainment, transportation, banking, healthcare, and of course in all business activities. And, in all these applications, the software is very complex. Millions of lines of code written by hundreds of programmers are integrated into one package. If there is a single mistake committed by one programmer, it may lead to chaos. Therefore, it is the software that has now become the lifeline of systems. The total money spent on computer hardware and software in the world is exceeding US$ one trillion every year. In the US alone, the total annual expenditure on software projects is US$60 billion. And, as more and more countries and people become computersavvy, the expenditures on software will go up. As automation has brought in revolutionary changes in the quality of life, the demand for more software-driven services and products is increasing. The ubiquitous mobile phone is an example. Two decades ago, the mobile phone was only for the elite of the society. Today, it is a necessary gadget in everyone’s hand. Today, mobile phone is a complex technology gadget with nearly two millions of code residing in it. As the users demand more and more services through their mobile phones, it is expected that the software will be about 10 million lines of code in the next few years. This indicates how the complexity of software is increasing day by day, as users demand more and more facilities in their existing services.

Chapter 1

2

1.2 Challenges in Software Projects The complexity of software projects is increasing tremendously. Organizations with operations in different locations in different countries need sophisticated software to manage their operations. For example, an organization like Wal-Mart needs to have efficient supply chain management software to ensure that every retail store gets the necessary material in time. As users demand more facilities for their existing services, the software becomes more complex. For instance, bank account holders no longer want to go to a bank during the working hours to do any transactions—they need ATMs, Internet banking, mobile banking, etc. making the banking software more and more complex. As crime continues to increase, software now needs to provide good security features. This is true for mobile phone software, banking software, e-commerce portals, etc. And to provide foolproof security is a very challenging task, as evident from the increase in virus attacks, credit card information being stolen from the Internet and the cyber-crime. Developing mission-critical software for a number of applications, such as bio-medical equipment, nuclear plants, transportation control systems, which demand zero-defect software, is highly challenging. As software is gaining more prominence, the demand for software professionals is increasing exponentially, but there has always been a shortage of quality professionals. As a result, employees jump from one organization to another, that too in the middle of the project leading to many projects getting into trouble halfway. Manpower attrition is one of the main challenges to be tackled in software organizations. As the demand for new software products and services increases, there is an immense pressure on the development organizations to deliver fast. The time pressure is so much that the organizations and people are unable to cope up with it. Unfortunately, in many cases, this is leading to the development of software full of bugs. Though we all want every software project to be a great success—good quality software delivered within the time frame and within the budget—it happens rarely. So, our aim should be to make it happen every time for every project. The challenges in software development are as follows: developing zero-defect software, meeting severe time constraints, increased complexity of the projects and shortage of skilled people.

1.3 Software Fiascos Software development is considered more of an art and less of science or engineering. To bring software development into the realm of engineering, a lot of research has been carried out during the last four decades. In spite of the research in finding out ways and means of developing quality software, the history shows that a large number of projects are executed very badly. We keep reading in the newspapers about the mobile phones

Introduction

3 that have been withdrawn from the market because of a software problem, about the credit card fraud, and so on and all these incidents are software fiascos. There are many stories in the history of software development that teach us very important lessons. Here, are some famous stories that the test engineers need to keep in mind.

1.3.1 Ariane 5 On June 4, 1996, European Space Agency launched the Ariane 5 satellite launch vehicle (Figure 1.1). Exactly 40 seconds after lift-off at an altitude of 3700 meters, the launcher exploded and became a ball of fire. The project, which was uninsured, costed US$ 500 million. What went wrong?

Figure 1.1 Ariane 5 Launch Vehicle In one of the embedded systems, a software exception caused the problem. A 64-bit floating-point number was to be converted into a 16-bit signed integer value. During one calculation, the converted integer value was more than the value that can be represented by a 16-bit signed integer. This caused an operand error. Yes, the software exception was handled, but badly. The specification of the exception handling routine was to indicate the failure on the data bus, to store the context in the EPROM and to shut down the

Chapter 1

4 processor. The processor was shut down, and this created the chaos. The mishap can be attributed to lack of enough testing as well as flaw in the software design, particularly in exception handling. This error is known as 500 million dollar software error in the software literature [Jezequel, 1997]. Interestingly, the software used in Ariane 4 was reused in Ariane 5, but Ariane 4 was a successful launch. The only difference is that in Ariane 5, this calculation was not at all required! The failure of this project is a lesson for every software engineer—one line of code can cause irreparable damage. Ariane 5 project illustrates the dangers associated with leaving some portions of the untested code in the software. The test engineers need to ensure that each and every line of the code is tested before releasing the software.

1.3.2 Patriot Missile During the 1991 Gulf war, an Iraqi Scud missile hit an American army barrack killing many soldiers. An American Patriot missile (Figure 1.2) could not track and intercept the Scud missile. The reason: a software bug in the Patriot missile.

Figure 1.2 Patriot Missile Whenever the tracking system of the Patriot was switched on, it measured the elapsed time from an internal clock in tenths of a second. This value was multiplied by 10 to get the time in seconds. The calculation was performed using 24-bit fixed-point registers.

Introduction

5 Hence, the value of 1/10 stored in a register was truncated to 24 bits, whereas the exact value is 0.000110011001100110011001100…..When the value is truncated, an error is introduced and this error accumulated over a period of time. The Patriot missile was switched on for about 100 hours, and so the accumulated error was 0.34 second. Scud traveled with a speed of 1,676 meters per second, i.e. or more than half a kilometer in 0.34 seconds. As a result, the Patriot could not intercept the Scud. This incident indicates the care that needs to be taken, while designing and testing mission-critical systems. The Patriot Missile example shows how a small software bug—accumulation of error due to rounding off a number—can lead to a catastrophe.

1.3.3 Mars Pathfinder The NASA mission Pathfinder landed on Mars on 4 July, 1997. The mission was successful—a perfect launch, a perfect landing on Mars and a lot of important data was sent to Earth. After a few days, the computing system started resetting intermittently and the important data was not being received on Earth. The on-board computer used a realtime operating system. Using an exact replica of the computer on Earth, the debugging was done to find the fault. Three tasks were running on the system—one with high priority, one with low priority and one with medium priority. Intermittently, the task with medium priority was running for a long time, while the task with high priority was waiting for the task with low priority to complete. The task with low priority was not getting the CPU time because a higher priority task was running. Meanwhile, the watchdog timer, suspecting a deadlock, used to reset the processor. This problem was due to a bug in the inter-task communication module. Of course, the story ended in a positive note, as the engineers were able to upload a modified file on to the Pathfinder. The story of the Pathfinder teaches us many lessons in software development. While doing the simulation studies, the engineers noticed such a problem before the launch of the Pathfinder, but they thought that the problem was due to a hardware glitch as it was observed only a very few times—we should not set aside the occasional problems encountered even. The engineers on Earth could analyze and rectify the bug only because the debug code was also running on the computer of the Pathfinder—sometimes the debug code is very important, we should not think that debug code is only for development, it is very useful for maintenance as well. Mars Pathfinder is an excellent example to understand the complexity involved in testing real-time systems. The bad things are the bad design and the approach taken by some engineers when a problem was encountered while testing the software in the laboratory. The marvel is that the problem could be solved without someone flying to the Mars to upgrade the software in the field!

Chapter 1

6

1.3.4 CT Scanner Getting a CT scan (Figure 1.3) done in a hospital is a very simple and painless process. Really? Think again!

Figure 1.3 CT Scanner In a Canadian hospital, suddenly a patient undergoing the scanning died. The machine was rechecked thoroughly and then again the scanning started. After scanning of a few patients was completed, another patient died. Randomly, patients were dieing and after the death of more than 20 patients, the defect was identified. A glitch in the hardware was resetting the software. During the short time of that resetting, the patient was exposed to large amounts of radiation because of lack of control on the radiation. As a result, the patient used to die. Next time, when you get a CT scan done, pray to God that the software should have been developed and tested by those engineers who read this book, the people with quality consciousness. The CT Scanner example shows how difficult it is even to catch a defect. A hardware defect gives rise to a software defect and that too, it occurs intermittently. In safety critical applications, compromising on quality results in loss of life even.

1.3.5 The Great Bank Robbery Long ago, in a bank automation project, a development engineer and a bank employee together played a simple trick: while calculating the interest, truncate the value to two digits and deposit the remaining value in their own account. For example, suppose interest is calculated at 3.5% on a deposit of US$ 124.25. The interest will be 4.34875. Out of this amount, $4.34 would be credited to the actual account holder’s account and the balance $0.00875 to their account. A simple trick—since the amount is very small, no

Introduction

7 customer will complain to the bank. But the amount would be substantial when millions of transactions take place every month. The crime could not be noticed for a few years, it was detected only when their account was suddenly closed fearing that they would be caught. This is a very interesting case study for those who write the specifications of the software—whether you should truncate or round off a number. Test engineers also need to appreciate the interesting aspect of this episode—you need to test the software for what it is supposed to do, and also for what it is not supposed to do! This great bank robbery indicates that software engineers also can do great robberies! And, how do you find that such a crime was committed? Only when each and every line of code written by the developers is reviewed.

1.3.6 FBI Virtual Case File Federal Bureau of Investigation (FBI) of the US Government initiated a US$ 170 million project in 2001. The aim of the project was to develop a data warehouse that stores the multi-media information gathered by the intelligence agents all over the world, which provides the necessary intelligence for future investigations. The software component of this project was worth US$105 million and 700,000 lines of code were written. After three years of development, the project was officially closed in April 2005 and declared as Dead on Arrival. This is one of the most publicized and analyzed failed software projects in the recent history. For the US Government, it was a real unfortunate project—after 9/11 terrorist attacks, the most vital tool required for FBI for providing national security could not be delivered. You can refer to the September 2005 issue of IEEE spectrum to read the details of the Virtual Case File project. This issue also carries another article [Charette 2005] giving Hall of Shame a list of projects that failed, such as the ERP system of HP, supply chain management systems of Nike and K-Mart, CRM upgradation software of AT&T Wireless, etc. The field of software is full of such stories. Refer to [Glass 1999] for the lessons to be learned from computing calamities. The calamities include products and projects that failed and organizations that failed. Ariane 5, Patriot missile, Mars Pathfinder, CT Scanner, the great bank robbery, and the Virtual Case File stories teach us very important lessons about the intricacies of software testing. In our professional career, we learn more from failures than successes! It is also equally important to learn from others’ failures. However, very few failures are discussed in public—unfortunately, we like to hear about successes not failures.

Chapter 1

8

1.4 Reasons for Software Failure As [Charette 2005] writes in his article, “we waste billions of dollars each year on entirely preventable mistakes”. Every project is taken with great expectations and during execution, somewhere somehow things go out of hand and the projects end up as a big failure. If you start listing the reasons for failure of projects, you will end up with 100’s of reasons. However, the reasons can be attributed to mainly the following categories:

y People: Ultimately, every problem is attributed to the people. Lack of quality

consciousness, lack of proper communication, lack of inter-personal skills, personal idiosyncrasies and lack of professional ethics are the main reasons.

y Technology: Sometimes, the focus shifts from the user requirements to technology.

Excessive fascination towards introducing new technologies leads to lot of problems while implementation.

y Time: Due to market pressures, organizations try to compress the development

time so much that the targets become unrealistic, leading to overworked engineers who in later stages of the project somehow want to get rid of the project rather than bring out a quality product.

y Money: At the time of initially estimating the project cost, we commit mistakes,

particularly when we are working on new technologies. It is simply, because at the beginning we do not know enough the details to estimate correctly. Once, a budget is given, the project must be completed within that stipulated budget; else, it is declared as a failure. The reasons for failure of software projects can be attributed to people-related problems, technology-related problems, time-related problems and moneyrelated problems. A project is considered as a failure if quality software is not delivered or if the project is not completed within the time frame (time overrun) or if the project is not completed within the budget (cost overrun).

1.5 What is the Solution? Statistics reveal that nearly 74% of the software projects could not be completed within the time frame or within the budget or both. Nearly 20% of the projects are abandoned halfway. The problem can be due to the following:

y Wrong estimation before starting the project, y The project could not be managed and completed within the estimated time and budget.

Introduction

9 Many a time, the project is delivered within the time and budget, but compromised on quality, which is worse.

The software engineers and project managers have two choices—to live with such a painful process or to develop a process whereby the software development will be more of fun and challenge. To make software development a fun-filled challenge; a creative activity is needed that contributes to better quality of life, an intellectually satisfying job that makes the world a better place to live in. There is only one solution for this, a systematic and disciplined approach to software development. Managing software projects, in a highly competitive environment with fast changing technologies and shortage of skilled workforce, has become the biggest challenge for all software organizations. The project team has to make every project a success. But then, what are the secrets of success?

y Every employee in the organization should be committed to software quality assurance.

y The product must be tested thoroughly. y Every individual must follow a code of ethics and professional practice. To make software projects successful, the three important mantras are: a) software quality assurance b) software testing and c) code of ethics and professional practice.

1.5.1 Software Quality Assurance If we have to deliver quality software, as per the customer requirements, we need to be quality conscious throughout—from the beginning of the project to its completion. We divide the software development activity into a number of phases such as obtaining user requirements, design, coding, testing and maintenance. We can deliver a quality product to the customer only when we are conscious of quality in every phase of development. And this quality consciousness must be there in each and every person associated with the project—from the CEO to the programmers. Even one or a few persons take it easy, quality will be compromised. Software Quality Assurance (SQA) addresses the various mechanisms to be implemented to prevent defects in the software. SQA encompasses all phases of development, it is applicable to all types of projects and it addresses all the people associated with the project including the management. Generally, in software development organizations, there will be a separate SQA department. The persons working in this department will check whether the development teams are following the quality norms defined in the organization. In small organizations, there may not be a separate department, but at least one or few people are assigned the SQA responsibilities.

Chapter 1

10 To develop quality product, quality has to be ensured in every phase of development. Software Quality Assurance addresses the various mechanisms to be implemented during different phases of development.

We will present an overview of SQA in Chapter 2.

1.5.2 Software Testing Even though we ensure quality at every stage of development, we need to put special emphasis on the testing phase. In spite of the best efforts of SQA engineers and development engineers, it is likely that there are defects (many of them!) still waiting to be uncovered in the software. In the testing phase, we need to rigorously test the software and find out as many bugs as possible, so that, we can confidently deliver a good quality product to the customer. Because testing is the most important phase in development, organizations are setting up separate testing groups whose job is only to test the software developed by different groups. The test engineers specialize in testing. And, if you think it is very easy to point out the other guy’s problems; think again, testing is not that easy. Testing is a highly specialized activity. The testing professionals need to have special skills and they need to learn many tricks of trade. Every organization is now giving a lot of thrust to the testing team. How many people are involved in testing, of course, differs depending on the type of project. In Microsoft Corporation, the ratio of development engineers to test engineers is 1:1. In NASA Goddard Space Flight Center, the ratio is 1:7, i.e. for every development engineer, there will be seven test engineers. For mission-critical applications involving safety and security of people (e.g., space, medical, nuclear, transportation etc.), the testing teams should be pretty big. In NASA Goddard Space Flight Center, the ratio of development engineers to test engineers is 1: 7, i.e. for every development engineer, there will be seven test engineers. If quality is of paramount importance, organizations must invest on test engineers.

1.5.3 Code of Ethics Some software engineers, out of ignorance, do many illegal things. For example, you may come across an engineer who sends the code written by him to a friend working in another organization. This person is not aware that the code written by an employee is the intellectual property of the employer and it cannot be shared with others just like that. This ignorance may cost him his job. Some software engineers, out of fear towards their employer, do unethical things. Suppose, you are testing software for a medical equipment that will be used in intensive care unit of a hospital. The software still has bugs, but the manager asks you to declare

Introduction

11 that the software is OK. What will you do? If you say the software is OK, you may kill a patient, and if you say the software is not OK, you may lose your job. In such situations, the code of ethics provides you the guidelines based on which you can take a decision based on what you think is morally correct. Every software engineer has to realize that he/she significantly contributes for betterment of society and for improving the quality of life. So, he/she has a responsibility towards society and the software developed by him/her should be of high quality— otherwise, it may result in loss of money or loss of time or even loss of lives. Keeping in view the important role played by software engineers as professionals, ACM and IEEE developed a code of ethics and professional practice. Every software engineer, involved in development, testing, management or related functions must follow this code. This code helps you in understanding what is ethical and what is unethical in your profession. This code of ethics is for all persons involved in all aspects of development—that includes programmers, test engineers, managers and also those who teach software. This code is discussed in detail in Chapter 9. Many engineers are unaware of code of ethics for professionals. In every organization, the HR managers need to make it a point that this code of ethics is included in the training programs for. According to Oxford dictionary, ethics means study of moral principles. Code of ethics the employees provides the guidelines based on which you can decide what is morally correct.

1.6 Software Testing Professionals Software testing professionals are now becoming glamorous people. Because they do a professional job in a specialized area, they are in great demand. They command lot of respect (even by the development engineers) and make a lot of money. However, to achieve this status, you need to really work hard. As shown in Figure 1.4, the test engineers have to interact with many other groups to carry out their work. So, the fundamental requirement of a test engineer is to have good inter-personal skills.

Chapter 1

12

Quality Team

User Requirements Analysts

Test Engineers

Developers

Project Leaders Project Manager

Figure 1.4 Test engineers’ interactions

1.6.1 Skill Sets for Testing Professionals The various skills you need to acquire to become software professional are:

y A good understanding of software quality assurance and quality management standards such as Capability Maturity Model Integration (CMMI).

y A thorough knowledge of testing process, testing types and methods, designing and executing tests.

y Hands-on expertise in using popular testing tools. y Knowledge in an application area (such as financial accounting, business applications, telecommunications, networking, security, etc.).

y Excellent communication and inter-personal skills with ability to effectively inform bad news (that the software has bugs).

y y y y

Good documentation skills to document test results and test reports. Good listening skills, to listen to the members of other groups. The art of learning fast new techniques and technologies. A good understanding of the “code of ethics and professional practice”.

Introduction

13 Many young engineers think that communication skills mean the ability to talk well. But, in addition to talking, listening is equally important. Many people are very fond of talking and become restless when it comes to listening.

1.6.2 Tasks Handled by Testing Professionals. Many young software engineers think that testing is not a creative job. Contrary to this popular belief, testing is a very creative and challenging work. The test engineers need to understand the application well, study the software functionality in detail to find out where the bugs are likely to occur, study the code to ensure that each and every line of code is tested, create test cases in such a way that testing is done rigorously to uncover the hidden bugs and also ensure that the software is usable and reliable. All this has to be done very fast—the testing engineers are given very less time in the whole life cycle of product development! To take up testing of a software product, the test engineers need to do the following:

y Understand the application domain: The software may be a Client/Server based

database management system, or a telecom software, or data communication software, or embedded software, or process control software, a web site, etc. In your career as a test engineer, you may have to test software of various application domains. Before starting the testing, you need to understand the application, from an end-user point of view. You need to be a very fast learner of the application domain.

y Learn about the development environment: Based on the application and the end

user needs, the development environment varies—the operating system may be windows, UNIX, Linux, or Solaris; the programming language may be C, C++, or Java; some special tools would have been used to develop the software such as database engines, cross-compilers, etc. You need to have flair to work on different development environments.

y Learn how to use the application: The test engineer has to do assume that he/she

is the end user and run the application to get a feel of how the software works. Only then, he will be in a position to generate the test cases and also create automated test procedures.

y Study the source code: If the test engineer is asked to validate the source code also,

he needs to study the source code based on which the test case generation has to be done.

y Study the requirements specifications document: The requirements document

gives the functional, performance and reliability requirements of the software. The software has to be tested for all these requirements. You need to be eagle-eyed and look for detail while studying the specifications and check whether each requirement can be tested or not. If the SRS document says “the transaction

Chapter 1

14 response should be very fast”—it does not mean anything! You need to tell the person who wrote the specifications that the response time should be indicated, say 10 seconds. Then only you can test this performance requirement.

y Study the acceptance test procedure: For every project, an acceptance test

procedure document has to be developed. This document gives the test plan based on which the customer will validate the software. The test engineers need to carry out the testing as per this document. This document has to be prepared by the development team and the testing team in association with the customer representative.

y Test innovatively: Test the software in minimum possible time to uncover maximum possible bugs. This can be done by designing tests innovatively and using testing tools efficiently. You are always constrained for time. You need to find the defects as fast as possible and that is the great challenge.

y Report the test results: Report the test results to the development team. This has to

be done very carefully, because no developer would like to hear that there are bugs in his/her software.

y Contribute to the organization’s growth: The bottom line is to help your

organization to grow. Your contribution as a test engineer to the quality of the software would help in customer satisfaction, which in turn leads to more business for the organization.

y Be proud of your work: Many test engineers feel that they are doing only testing

and not coding. Testing is as respectful a job as coding. In fact, organizations are giving equal importance and value to the test engineers as development engineers. Only when you feel proud that you are involved in an important work, you can do a good job. The test engineers need to have an eye for the details, in other words, they need to be eagle-eyed. But then, they need to offer constructive criticism to improve the quality of the software rather than finding fault with the developers.

The next chapter presents an overview of software quality assurance. A good understanding of this chapter gives a solid foundation for test engineers.

Summary y Software has become the lifeline of systems used in all walks of life—business, education, entertainment, financial services, transportation, health care, etc.

Introduction

15 y Every software engineer has to strive to develop zero-defect software as one defect may lead to loss of millions of dollars, loss of life, etc.

y To develop quality software with constraints of time and budget is very challenging.

y Ariane 5, Mars Pathfinder, Patriot missile, CT Scanner, the Great Bank Robbery, and Virtual Case File incidents indicate that small errors can create chaos.

y The secrets of success for software projects are software quality assurance, software testing and software ethics.

y Software quality assurance specifies the mechanisms to be followed throughout the development of software. It encompasses all the phases of development.

y To produce quality software, organizations are now having separate testing teams, with a large workforce specializing in testing.

y Test engineers need to interact with user/requirements analysts, developers, QA team members and project managers effectively.

Review Questions 1. The Ariane 5 crash occurred due to: a) Conversion of a 64-bit floating point number to a 16-bit integer b) Accumulation of truncation error c) Wrong assignment of priorities to tasks in real-time operating system d) None of the above 2. In Patriot missile, the mishap took place due to: a) Wrong assignment of priorities to tasks in real-time operating system b) Conversion of 64-bit floating point number to a 16-bit integer c) Accumulation of truncation error d) None of the above 3. The problem in Mars Pathfinder occurred due to a) Conversion of a 64-bit floating-point number to a 16-bit integer b) Accumulation of truncation error c) Wrong assignment of priorities to tasks in real-time operating system d) None of the above 4. To deliver quality software, which of the following is a must? a) Software quality assurance

b) Testing

c) Ethical values in the team members

d) All the above

Chapter 1

16 5. In NASA Goddard Space Flight Center, the ratio of development engineers to test engineers is: a) 1:1

b) 1:2

c) 1:6

d) 1:7

6. To produce quality software, it is advisable to have a separate testing team a) True

b) False

7. SQA does not encompass all phases of software development: a) True

b) False

8. Cost overrun implies that a) the project is not within the budget expected by the customer b) the project is not completed within the stipulated budget c) the project could not be taken up because of high cost d) it is not worth taking up the project because it is of small value. 9. An error in one line of code can make the system fail a) True

b) False

10. Faults in software systems occur always due to code. a) True

b) False

Chapter

Software Development Life Cycle and Quality Assurance

2

Chapter Objectives y y y y y y y y

Various stages in software development Importance of software engineering Criteria for the success of a software project Software Quality Triangle Software development life cycle models Overview of software quality assurance Software product and process quality metrics Software process quality models and overview of CMMI

During the first two decades of computing, software development was being carried out on ad hoc basis without any methodology. This led to lot of difficulties in maintaining the software. To overcome these difficulties, software engineering emerged as a discipline. In this chapter, we present an overview of software engineering. We will cover the entire software development life cycle and then study the various aspects of software quality assurance. We will also discuss the various models used for software development. The difference between product quality and process quality is also explained and an overview of Capability Maturity Model Integration (CMMI) is presented.

2.1 The Software Crisis During the initial days of computing, software was largely developed by small groups of people. When this software was to be maintained by another set of people, it used to be a nightmarish experience. When large commercial projects were to be handled, it used to be an almost impossible task to keep the time and budget under control, and projects used to get delayed and many projects were abandoned halfway.

Chapter 2

18 During these years, software development was carried out in assembly languages and procedural languages such as FORTRAN, COBOL, BASIC and LISP.

Programming was mainly an art—confined to a set of highly specialized persons who developed and maintained the software for years. There was no systematic procedure for the software development; everything was done on ad hoc basis. This ad hoc approach led to the software crisis.

2.2 The Birth of Software Engineering In the year 1968, a conference sponsored by the NATO Science Committee addressed the software crisis that was prevailing in those days. The outcome of the deliberations of that conference was software development is not an art, it has to fall in the realm of engineering. The term software engineering was introduced in this conference. It is nearly four decades since the birth of software engineering. During this period, a lot of research went on in making the software development an engineering discipline. Software engineering defines a disciplined approach to software development.

2.3 What is Software Engineering? In most of the software development organizations, there is no systematic methodology for the development. Software development is considered as coding, though coding forms a very small percentage of the total development effort. Before starting the coding, one has to systematically approach the problem. One has to understand the requirements (what the software is supposed to do), carry out the design, do the coding, carry out a rigorous testing and if the software is as per the requirements, release the software to the customer. Subsequently, if the customer wants some changes---refinements or enhancements---then the software has to be modified. Software engineering can be defined as systematic development of software. IEEE Standards [IEEE 1990] define software engineering as the application of a systematic, disciplined, quantitative approach to the development, operation and maintenance of software, i.e, the application of engineering to software. For developing a software product, the development process is divided into different stages. A convenient way of splitting the whole process is to divide the development into the following five stages:

y Specifications y Design y Implementation

Software Development Life Cycle and Quality Assurance

19 y Testing y Maintenance In addition, to ensure that the people are effectively utilized, management processes are defined, such as planning process, review process, monitoring process, etc. So, software engineering encompasses all activities of managing technology, money, people and time to deliver quality software products. We will discuss the details of software engineering in this chapter, covering both technical and managerial aspects. Software engineering is a discipline that deals with the systematic development of a software to a produce quality software. As per IEEE standards, software engineering is the application of a systematic, disciplined, quantitative approach to the development, operation and maintenance of software, i.e. the application of engineering to software.

2.4 Why Software Engineering? Every organization involved in software development is realizing that utmost importance is to be given to software engineering. Organizations are being assessed and graded based on the software engineering principles being implemented. If an organization has to survive and grow in the competitive environment, software engineering plays the key role. Why? Here are the reasons:

y To deliver quality product: To ensure that the software developed meets all the expectations of the customer is the most important goal for any organization, because a satisfied customer can enhance the image of the organization and bring in more growth. Software engineering helps in achieving this goal.

y Increasing competition in the software industry: As software industry is one of

the highest growth industries, the number of players has increased tremendously. To survive in a competitive market, organizations have to show their commitment to quality and also get their software development processes assessed for better visibility and software engineering is the medicine.

y High manpower attrition: Because of the shortage of highly skilled manpower,

every organization is facing the problem of manpower leaving in the middle of the projects. As software development is people-oriented, organizations have to define processes so that even if people leave in the middle of the project, the project will not be adversely affected. Software engineering contributes significantly in achieving this goal.

y Increasing development costs: Over the years, the cost of hardware is falling

gradually; thanks to the developments in semiconductor technology. Software engineering helps in reducing the development costs. On the other hand, the cost of

Chapter 2

20 software is growing drastically because of the rise in establishment and manpower charges, etc.

y Time to market: Because of increased competition, software products have to be

delivered as per schedule and one cannot afford to miss the deadline. The Just in Time release of the product is of utmost importance. Software engineering defines processes, whereby, time schedule can be kept under control.

y Fast changing technology: In software, the technology is changing very fast, so fast

that it becoming very difficult to cope up with the pace—for organizations and for people. To manage the change also, processes need to be defined so that the organization is not left behind in adapting new technologies.

y Sensitivity involved in software products: As society becomes more and more

dependent on software, particularly when used in life-saving equipment (such as medical equipment), utmost care has to be taken to ensure that the software is highly reliable and meets the safety requirements. Software engineering helps in achieving this goal as well.

y Increasing complexity of projects: The complexity of software projects is

increasing tremendously. As the world shrinks, the complexity involved in developing software for organizations having global operations, or satellite launching, or telecommunication software, is very high. Managing such projects involving hundreds or thousands of software engineers working for a few years is a gigantic task. Using an ad hoc approach to manage such projects will result in chaos. Software engineering is a must for such projects.

y Rise in subcontracting of software development and IT services: Many non-IT

organizations (such as service industries, hospitality industries, manufacturing industries whose main activity is not IT) are subcontracting their software development and maintenance work to software companies, as it will be costeffective for them as compared to having their own development staff. Subcontract management is crucial and has to be done in a systematic way. Software engineering helps in achieving this.

y Off-shore development of software: Many developed countries are outsourcing

their software development work as the development costs are comparatively lower in developing countries like India, China, Malaysia, Thailand, Indonesia, etc. Managing the offshore development work (when things are not directly under the control of the prime contractor) is not an easy task. Software engineering again comes to the rescue.

y Increasing the productivity of people: As the manpower costs go up, the

manpower has to be utilized effectively, continuous effort has to be made to increase the productivity of the individuals and teams. Software engineering provides the solutions.

Software Development Life Cycle and Quality Assurance

21 Software engineering is a must for every organization—big or small. Even in small organizations, the days when development is carried out on ad hoc basis are gone; for an organization to grow, software engineering is the lifeline. To summarize, today every business house has to aim for building High quality software products at competitive prices just in time and there is no escape. To achieve this organizational goal, there is only one method—to follow software engineering principles. Every organization has to develop a goal oriented cybernetic system (i.e. a system which is constantly improved based on the past experience) whereby the organization can achieve the above goal and also increase its productivity as well as productivity of its employees. The goal of every organization and every employee should be to develop high quality software products at competitive prices just in time. Software engineering helps in achieving this goal.

2.5 Is the Crisis Overcome? No, not yet. Software projects continue to be crisis-oriented. Software project management in many organizations is still amateurish and very few tools or techniques are used even for managing very large software projects. Software engineers, in spite of learning the theory of software engineering during their school days, hardly practice it. Documentation continues to be considered a ‘bad’ job as a result of which if a person leaves in the middle of a project, it is difficult for others to continue the work. Due to lack of methodology in writing the source code, it becomes very complicated to maintain the code after it is delivered to the customer. Due to lack of awareness about professional ethics, many software engineers are committing white-collar crimes. Because of the cumulative effect of all these (and many more such) factors, the crisis continues. A strong commitment to software engineering principles by the top management, software project managers and engineers is required so that software projects do not get into crisis. Every employee of a software development organization has to strive to make every software project a success. But then…

2.6 Measuring Success of Software Project Managing software projects, in a highly competitive environment with fast changing technologies and shortage of skilled workforce, has become the biggest challenge for all software organizations. The project manager has to try to make every project a success. But then, what is a successful software project? We will now define the criteria for the success of a software project.

Chapter 2

22

2.6.1 Criteria for the Success of a Software Project When do we say that a software project is successful? Success is a perception. After a project is completed, you can ask the team members and also some persons outside the team whether the project is a success. Do not be surprised if you get different answers. It is very difficult to get an objective answer to what a successful project is. The four criteria for the success of a project are:

y y y y

The software must meet all the quality requirements. The software must be developed within the time frame. The software must be developed within the budget. The relations amongst the team members should be cordial during the project execution and after the project is completed.

We will study each of these criteria in detail. A project can be termed as successful only if the software meets the quality requirements, the project is completed within the time frame and within the budget, and all the team members maintain a cordial relation during the execution of the project.

Quality The most important requirement of a software product is that it should meet the requirements of the customer. In addition to the user requirements, the management of the development organization may put additional requirements to ensure that the product is developed at a low cost and also to ensure that the software can be maintained at a low cost. If the management wants to modify customized software to make it a generic product at a later date, additional requirement may be put, such as portability. In general, the characteristics of a software product can be divided into:

y Operational characteristics. y Transition characteristics. y Revision characteristics. The characteristics of a software product can be divided into operational characteristics, transition characteristics and revision characteristics.

Software quality triangle, shown in Figure 2.1, depicts these characteristics. Operational characteristics specify the requirements during the operation/usage. Transition characteristics specify the requirements for its usage in other hardware/operating system environments. Revision characteristics specify the requirements for making the changes to software easy.

Software Development Life Cycle and Quality Assurance

23 Correctness

Portability

Usability/Learnability Integrity

Reusability

Efficiency

Interoperability

Reliability Safety Security

Transition

Operation

Revision

Maintainability

Extensibility

Testability

Scalability

Flexibility

Modularity

Figure 2.1 Software quality triangle The operational characteristics are:

y Correctness: The software should meet all the specifications/requirements given by the customer/users.

y Usability/Learnability: Effort/time required to learn how to use the software, should be minimal.

y Integrity: The software should not have side effects (like another application does not run when the software is invoked).

y Efficiency: The software should make effective use of the storage space, execute as per the desired timing requirements etc.

y Reliability: The software should be defect-free and should not fail during the operation.

y Safety: The software should not be hazardous to environment/life. y Security: The software should not have ill effects on data/hardware.

Chapter 2

24 Some people use the words quality and reliability interchangeably. Note that reliability is one aspect of quality.

The revision characteristics are:

y Maintainability: It should be easy to maintain the software, i.e. time/effort required to remove defects in the software should be minimal.

y Testability: It should be easy to test the software, i.e. the time/effort required to test the software should be minimal.

y Flexibility: It should be easy to make modifications to the software, i.e. the time/effort required for making modifications should be minimal.

y Scalability: It should be easy to increase the performance of the software if the application demands it.

y Extensibility: It should be easy to enhance the functionality of the software if the application demands it.

y Modularity: If the software is divided into separate independent parts that can be modified, and tested separately, it has high modularity.

The transition characteristics are:

y Portability: Ease with which software can be transferred from one platform to

another platform (for example from Windows XP to UNIX) without changing the functionality.

y Reusability: If portions of the software can be used in some other application with little or no modifications, the software is said to be reusable.

y Interoperability: Ability to make the software exchange information with another system and make use of the information transparently.

Ideally, every software product must have all the above quality characteristics. However, depending on the application, only a subset of these characteristics may be required. Before starting the development of a software product, a requirements specification document is written clearly indicating all the quality requirements. After the product is developed, the software has to be tested to ensure that it meets the requirements of the customer and also the internal requirements. To differentiate between the two types of testing, the two terms validation and verification are used. Validation is used to check whether the software meets the customer expectations, wheras Verification is used to check whether the software conforms to all the specifications. The customer is not bothered about the reusability or portability of the software as long as it meets his requirements. So, the customer independently carries out an acceptance testing of the software.

Software Development Life Cycle and Quality Assurance

25 Validation is used to check whether the software meets customer expectations. Verification is used to check whether the software conforms to all the specifications. Suppose the customer wants you to develop a software package on Windows XP platform. In such a case, you need not worry about the transition characteristics at all and focus only on operational and revision characteristics.

In many organizations, it is very common to deliver the software to the client, knowing fully well that there are many bugs, mainly to meet the deadline given by the client. Compromising on quality leads to increased maintenance costs. Generally the maintenance cost is much higher than the development cost. So, to make the software project a success, the first and foremost important requirement is the quality of the software. At the time of studying the requirements, the software developers need to analyze the various characteristics given in the quality triangle and decide which characteristics are applicable to the product under development. Test engineers need to test the software as per the specified requirements.

Time Factor To deliver the product within the time frame without compromising on quality is another criterion for the success of a project. Many clients are ready to pay more money if necessary but they do not like to compromise on time—after all time is money. But then, there is a problem. While giving the project proposal, the development time is estimated to be, say, six months. But, the client wants it in four months. So, to grab the order, in most cases, the project manager will agree to complete the project in four months. Once the commitment is made to the client, it is the manager’s responsibility to see that the project is completed within four months, and then only the project is a success. Here, the perceptions differ—the manager may argue that even if the project is completed in five months, for him it is a success because it is within his initial estimate. But the fact remains that the customer is the ultimate reference. More than the project team’s perceptions, the perception of the customer is important. A common strategy followed by many software development organizations is to commit to the client for a specific time frame, knowing well that the commitment cannot be fulfilled. Once the client awards the project, the client has no other alternative except to tolerate the delay. To ensure that one does not become a victim to such tactics, the client generally puts a late delivery clause, such as 0.5% of the project cost as penalty for delay per week. The project manager has to be forthright in his/her commitments. If the project cannot be completed within the time frame stipulated by the client, it is better not to accept the project at all. However, if the time frame is reduced only marginally, the manager has to find means and ways of completing the work and honoring the commitment.

Chapter 2

26 Unfortunately, in majority of software projects, extremely tight deadlines are set; resulting in tension for employees, fights between managers and engineers, arguments between client and developer, etc. You can read the fascinating book Death March by Edward Yourdon—this book describes the social and psychological impact of such tight deadlines. The client may not agree to the time frame suggested by you. When he reduces the time frame and if you agree to it; you must complete the project by the date committed by you.

Project Budget Project costing is an extremely complex process. While giving a project proposal to the prospective customer, it may not be possible to foresee each and every item to be costed. The initial estimate for the project cost may turn out to be erroneous for many reasons:

y The initial estimate of workforce may be incorrect. Initially you may estimate that only four persons are required; but after the commencement of the project you may realize that six persons are required.

y The initial estimate of time may be incorrect. If the project is delayed, you have to bear the additional burden of the salaries and overheads. In addition, you may have to pay penalty for the late delivery.

y There may be unforeseen technical problems because of which additional personnel or external-consulting services may be required.

y There may be changes in requirements due to which additional effort is needed. y Due to oversight, costing under a particular head might not have been accounted. y There may be changes in government rules as a result of which the cost would go up (e.g. increase in sales tax).

y Staff leave in the middle of the project, because of which the schedule is likely to be affected.

The reasons can be many more. A meticulous planning and foresight are required to do the project costing. As experience is the best teacher in project costing, the project manager has to document all the factors that affect the project cost while executing each and every project. The process of estimation needs to be improved continuously. Another important aspect in estimation is the identification of risks associated with a project. During project proposal preparation itself, the manager has to analyze the likely risks and the corrective action to be taken and account for such costs as well. For instance, it is now common in most organizations for people to leave in the middle of the project. The manpower attrition is a risk item—the corrective action is to have backup personnel, i.e. put additional personnel from the beginning of the project itself. To reduce the effect of unforeseeable costs (such as taxes), the agreement with the client has to be

Software Development Life Cycle and Quality Assurance

27 worded in such a way that the interest of the developer is protected (e.g. Taxes as applicable at the time of delivery). When the project proposal is submitted, the customer may negotiate on the project cost. The manager may reduce the cost by cutting down the profit margin, but not to the extent that execution of the project results in a loss. Sometimes, during negotiations with the client, the manager agrees to reduce the cost drastically with the hope that he will cut the expenditure and it is likely that he will be in for an unpleasant surprise! With the above-mentioned factors that affect the budget, and to carry out the development within the budget is a challenging task. The software has to be developed within the planned cost of development. If there is an additional expenditure, it eats into the profit margin, which is not acceptable to the senior management. Many projects result in cost-overrun because the initial estimates are not correct. Experience is the best teacher for cost estimation.

Maintaining Relationships The software project manager has to ensure that cordial relations are maintained amongst all the persons involved in the project throughout the development period. Sometimes, the project is completed as per schedule within the budget and quality software is delivered, but the relations are strained. The manager has to ensure that good relations are maintained between:

y y y y

Project manager and the team members Project manager and the top management Team members and the client/end users Development team and other teams such as quality team, test team etc.

It is a very challenging task for the manager to take all these groups of people along with him during the execution of the project. The most important element in managing the relations is expectations management—how do you manage the expectations of these groups during the execution of the project? It is a complex task and many people opine that software project management is more of personnel management than technical management. Consider a project, which involves development of a software product that will take the whole world by storm. Suppose the top management calls the project manager and tells him that the product has to be developed in just a few weeks. The project manager says ‘yes’ knowing well that it is an impossible deadline, but then he feels that the top management does not like to hear a ’no’. The project manager starts the development and then when the deadline approaches, he drops the bombshell that another few weeks are required. This is an example of a commitment that cannot be fulfilled. As a result, tempers are lost, sometimes jobs are lost, careers are ruined and the software

Chapter 2

28 development environment turns into a battlefield, which leaves many dead. Is it worth working on such projects? Many project managers do not inform the clients the correct status of the project, but keep on telling that “everything is fine, we will deliver as per schedule”. And, finally when the actual day of delivery arrives, the customer is informed that it is likely to take a few more weeks. This is a wrong strategy because raising the expectations and suddenly giving a shock is very dangerous. We need to tell all the persons concerned—the client and the senior management—the reality of the development status as well as the issues related to quality of the software. Similarly, the manager has to get the right inputs from the engineers and make the engineers open up and express their opinions so that they give realistic dates for development and also the realistic picture of the quality of the software. Any issues related to the quality have to be resolved through discussions with the quality team and the test team rather than fighting endless battles. It is very common during software project execution, for the project managers to shout at the engineers, the top management to shout at the project manager, the client threatening the project manager that he will cancel the order and so on. These unpleasant situations are due to lack of a process-oriented approach to software development on the one hand, and due to improper relations management on the other hand. Though relationship management is one of the most important tasks of a project manager, it is not given enough emphasis. But, one has to remember, a project can be considered a success only if at the end of it, everyone associated with the project is happy to have been associated with the project. Software development is a team effort and most of the problems in software projects can be attributed to people. For the success of a project, team spirit is the most important requirement in every individual.

2.6.2 Client’s Expectations During discussions with the client, invariably every client will make his expectations very clear:

y High quality y Low cost y Deliver fast It is extremely difficult to meet all these expectations.

Software Development Life Cycle and Quality Assurance

29 Many projects, particularly many government projects get into trouble because of these expectations—they want high quality software at unreasonable time and the order will be given to the organization that quoted the lowest price. The client can be asked to pick up any two of the above three requirements, not all three at a time! Pick up any two, not all at the same time.

The client always expects: high quality, low cost and fast delivery. Tell the client to pick up any two. If you get into the trap of accepting all the three, you are into great trouble.

2.7 Software Quality Assurance To deliver a quality product, quality has to be ensured in all stages of development. This process of ensuring quality at all stages is known as quality assurance. In Software Quality Assurance (SQA), the thrust is on process because software development cannot be done on an ad hoc basis. Without a process orientation, development leads to firefighting resulting in chaos. In spite of the stress that is given to the process by the management, many projects are executed without defining, let alone following, a process. For a project to be successful, process orientation is a must.

2.7.1 The Software Process As shown in Figure 2.2, to develop a product, a process is defined. The process takes the raw material as input and produces a finished product.

Raw material

Process

Finished product

Figure 2.2 The Process The process defines the various steps (or stages) to be followed to convert the raw material into a finished product. In case of software, there is no raw material (rather, the human brain is the raw material). As shown in Figure 2.3(a), the software process takes the problem definition as input and produces the software product.

Chapter 2

30

Problem

Software Development Process

Finished product

definition

(a)

Software Quality Assurance

Requirements

Design

Construction

Testing

Maintenance

Configuration Management (b)

Figure 2.3 Software Development Process As software development is a highly intellectual activity, it is very difficult to define precisely the software process. As shown in Figure 2.3(b), the process of software development is divided into various phases and in each phase, a set of activities is carried out to arrive at the end product. We can produce quality software only when thrust is given to quality in each phase. That is why, as shown in Figure 2.3(b), Software Quality Assurance (SQA) is a process that encompasses all the phases of development. The entire software process can be divided into as many stages as you like, depending on the complexity of the project. For illustration, five phases are shown in Figure 2.3(b). A process is the step-by-step procedure to convert raw material into finished product. Software process is the detailed procedure to convert a problem definition into working software product.

2.7.2 Phases in Software Development Life Cycle Each phase in the development process will have a defined input and a defined output. This divide-and-conquer strategy works well, as managing individual phases will be

Software Development Life Cycle and Quality Assurance

31 easier and monitoring will also be effective. The software development can be broadly categorized into the following phases:

y y y y y y y y

Market analysis/research Feasibility studies Software requirements engineering Software design Construction or Implementation Testing Maintenance Retirement

These phases are discussed briefly in this section.

Market Analysis/Research This phase is required mainly for generic product development. After a product is conceived, before embarking on the development, a thorough market survey has to be done. The objective of the market survey is to check whether the same or similar product already exists in the market, and if so, in what way the proposed product will have an edge over the existing products—in terms of features and cost. This survey should aim the following aspects:

y To define the product functionality. y To decide the operating environment for the software such as the Operating System, the user interface, network compatibility etc.

y The roadmap for the product, i.e. what the first version should be, and what features have to be added in the subsequent versions.

y How is the competition? If there are already a number of players, is it worth getting into the market? If yes, how the product should be placed vis-à-vis other products in terms of features and cost.

y Estimate the prospective customer base for the product keeping in view the competition.

y Estimate the cost per copy, based on the number of copies that can be sold and the development cost.

y Fix an approximate price for the software product and carry out a cost benefit analysis.

The entire market research/analysis results have to be recorded for future reference.

Chapter 2

32 In the case of software product development, a thorough market research/survey has to be done to gather the requirements of the proposed product and also to work out the end user price of the product.

Feasibility Studies In this phase, the feasibility of the project is ascertained. In addition to technical feasibility, the economic feasibility is also studied. If the project is not technically feasible, it is not worth taking trouble to go any further. If it is technically feasible, the cost-benefit analysis has to be done. If the project is profitable then the project can be taken up. Sometimes, even if it is not economically feasible, if there are indirect benefits, then also the project may be taken up. This is an important phase in which all the risks associated with the project have to be studied in detail and a decision is taken whether to go ahead with the project or not. The loss may be heavy if a project, which is not viable either economically or technically, is taken up. In addition to the monetary loss, there will be a loss of credibility and the future of the organization may be at stake. If the project is similar to an earlier project, then the manager can confidently take up the project. If the project is in a new area or if some stringent performance requirements are specified, then the risks associated with taking up the project need to be assessed thoroughly, and the impact of these risks are to be studied. In some cases, there may be a liability clause—if the project is not completed as per the specifications or if it is not delivered on time, the developer may be liable to pay heavy penalties. Hence, the manager needs to carry out a thorough feasibility study and take up the project only if it is viable. Feasibility study involves technical feasibility and economic feasibility. A project may be taken up even if it is not economically viable (i.e. there is no profit margin) if there are indirect benefits.

Software Requirements Engineering In this phase, the requirements of the client are obtained, analyzed, documented and validated. Utmost care has to be taken by the project team to obtain the requirements of the client to the highest degree of accuracy. As shown in Figure 2.4, the inputs to the requirements engineering process is the problem definition and the output is a validated Software Requirements Specifications (SRS) document.

Software Development Life Cycle and Quality Assurance

33 Requirements Elicitation

Requirements Analysis

Requirements Specifications

Problem definition

Requirements Validation

SRS document

Figure 2.4 Software Requirements Engineering Process The project team members have to discuss with the client/users in detail on the exact functionality of the software and also the performance and other non-functional requirements. These requirements have to be analyzed in great detail to check the real needs, whether the features can be implemented or not, etc. After that, the requirement specifications have to be documented and a document called Software Requirements Specifications (SRS) has to be written. After necessary interactions with the client, it has to be ensured that the SRS document is as per the requirements of the client, i.e. the document has to be validated. Some of the mistakes commonly made are:

y To start the development work without even understanding the requirements correctly.

y Not documenting the requirements and hence the requirements will be in the minds of some people who interacted with the client and when they leave the project halfway, the whole project will be in trouble.

y Whenever any changes are made to the requirements, not to incorporate them in the document as a result of which the changes made are not known to all the team members.

y To accept a change asked by the client without analyzing the likely impact of the change on the project development time and budget.

It is not practical always to expect that the specifications will be frozen and assume that there will not be any changes. To the extent possible, all the requirements have to be documented and subsequently if any changes are to be made, a specific procedure has to be followed to ensure that all the team members are aware of the changes. Also, whenever a change is to be made, the impact of this proposed change on the time frame, budget and quality are to be analyzed. Sometimes, an apparently small change in requirements may result in major changes in the design of the software. So, any change to be made to the specifications has to be discussed by the project team thoroughly and if the change turns out to be major in terms of the additional work to be done, then the terms of the contract (time frame and budget) need to be re-negotiated.

Chapter 2

34 The requirements engineering phase involves obtaining the requirements, documenting the requirements, validating the documented requirements and also defining the procedure for making the necessary changes in the requirements document. Test engineers need to study the SRS document thoroughly because the testing has to be done to ensure that the software meets all the requirements. The test engineers have to ensure that each and every specification is testable. Hence, involvement of test engineers is a must from the beginning of the project.

Software Design In this stage, the system design has to be carried out. The system consists of both hardware and software. The functionality of hardware and software are to be separated. This is a very complex activity, known as co-design. After that the detailed design of hardware and software are to be carried out. As shown in Figure 2.5, the design process takes the SRS document as the input and the output is a validated design document. System design Or top-level design

SRS document

Detailed design Or logic Design

Design validation

Validated design document

Figure 2.5 Software Design Process Software design can be further divided into system design or top-level; detailed design or logic design; and design validation. In system design, the various subsystems/modules are identified and their interconnections are worked out. In detailed design, each module design is done and the algorithms and data structures are identified. In design validation, it is checked whether the design will meet all the user requirements. During this stage, it is advisable to study whether different alternative designs are possible and if so, which is the best design. Unlike in other engineering fields, in software design, there are no fixed formulas. Software design is a highly intellectual activity involving exploration of different alternatives, studying the pros and cons of each alternative and choosing the best alternative based on some design criteria such as usability, performance, reliability etc. For user interface design, database design, realtime system design etc., different methodologies are to be followed and the process of design has to be documented. With the advent of object-oriented technology, object-

Software Development Life Cycle and Quality Assurance

35 oriented design is catching up. Using a standard methodology such as UML (Unified Modeling Language), the design has to be documented. The output of this phase is a design document giving the details of various modules and the details of logic and algorithms. Software design stage involves working out the architecture, design of algorithms and data structures and then documenting the complete design.

Construction or Implementation In this stage, the design is converted into code. As shown in Figure 2.6, the input to the construction/implementation process is the design document and output is the source code. Ideally, the design document prepared by the software architects should be given to the programmers and the programmers should be able to do the coding. To achieve this, a very detailed design document has to be prepared, which is not very easy. So, the programmers need to do the construction design—work out the details how to convert the low level design to code with the aim of reducing complexity and also keeping in view that changes to the code are likely [IEEE 2004]. The programmers need to constantly interact with the designers or the designers also need to actively participate in the implementation. A unit/component is the smallest piece of code that can be tested independently. Generally, as a part of coding itself, each component/unit is tested.

Construction design

Coding

Validated design document

Unit / component testing

Source code

Figure 2.6 Construction/Implementation Process The coding has to be done in such a way that the testing would be easier and later, the maintenance would be easier. The programming language to be used, the development tools to be used are decided at this stage, if not done earlier in the design stage or if not specified by the user. While doing the coding, enough documentation has to be embedded in the code for easy understanding of the code by others. This code has to be reviewed for clarity and for meeting the functionality of the software.

Chapter 2

36 If some code written earlier can be made use of for the present project, it can be reused resulting in reduced development time. However, enough care is to be taken while acquiring the code for reuse. The output of this stage is the code in a given programming language, with each of the units (smallest code segments) tested. The implementation process involves converting the software design into source code in a given programming language. During the implementation stage, source code developed for other projects may be reused by appropriate adaptation of the reusable code. The objective of coding should be to reduce the testing and maintenance time/effort, not just the development time. Nobody will be able to maintain your code if you give your boy/girl friend’s names as variable names.

Testing Testing consumes the highest amount of time in the development life cycle, however it is given the least importance by the development teams. The reason is that testing is difficult. The foremost thing in testing is to test the functionality of the software. In addition to functionality, the software has to be tested to ensure that it meets the performance requirements, the reliability requirements and other requirements, such as portability, learnability (ease of use) etc. As shown in Figure 2.7, the testing process is an iterative process. The source code is converted into an executable code and various test inputs are given. Test Input

Source code

Executable code

Test Output

Defect analysis

Figure 2.7 Testing Process For every test input, the output is analyzed to check whether the program is functioning correctly. If there is a defect, the defect is reflected in terms of a wrong

Software Development Life Cycle and Quality Assurance

37 output (or no output). The defect is analyzed, and the source code is modified and again testing is carried out with the test inputs. This process is repeated till the source code is defect-free (or almost!). In addition to testing the software in the laboratory, testing also needs to be done in the actual working environment (beta testing). Sometimes, depending on the requirement, other types of testing such as stress testing (to test the software at the limits of its performance specifications) have to be done. While testing the software, when defects are found, the source code has to be modified. Sometimes it may be necessary to modify the design or even the specifications. This makes software development an iterative process. A methodology has to be worked out so that the modifications to specifications, design, code, etc. are done methodically. For carrying out testing, the testing process is to be clearly defined, which involves development of a test plan, carrying out the testing as per the plan and documenting the test results in the form of a test report. The test plan has to specify the resources required for testing, the test tools to be used, the test cases (inputs to be given), the types of testing to be done, etc. The output of this stage is completely tested software, which meets the requirements specifications. Amongst all the phases of software development, testing phase has to be given utmost importance and it has to be ensured that the software is defect-free. The testing has to be done methodically using a defined process and a variety of tests have to be conducted on the software to see that the software meets the requirements specifications.

Maintenance Once the software is released to the client and the client starts making use of the software, maintenance phase is entered. The developer has to keep track of the feedback from the client and the cost of the maintenance effort. When the client reports a defect or when the client requests for a modification, the changes to the source code (and also the related documents, such as SRS, design document, etc.) have to be done systematically. The work products (SRS, design document, source code etc.) have to be kept under configuration control. As shown in Figure 2.8, many versions of these documents may need to be generated. Version 2.0

Version 1.0

Figure 2.8 Versions of a work product

Version 1.1

Version 1.2

Chapter 2

38 To start with, there will be version 1.0 from which version 1.1 will be generated. Version 1.1 may lead to two other versions—1.2 and 2.0. Unless a systematic procedure is followed to handle multiple versions, the work products will be unmanageable. Each version of the software source code has to be maintained with the required documentation. The maintenance process involves configuration management of the work products, keeping track of the feedback from the client and also the effort spent on the maintenance. The various documents generated during the course of software life cycle are known as work products. These work products include: SRS document, design document, source code, test reports, user manuals, etc. During maintenance phase, the software may undergo changes based on user feedback, or based on the bugs found out by the client/user. When the software source code is modified, correspondingly other work products have to be modified using configuration management procedure.

Retirement The retirement of the software is in the hands of the client—he may like to use for a number of years or just a few months, depending on the amount of money spent on the software. High budget software will be used for years or even decades; low-budgeted software may be used for a few months or a few years. For generic products, the retirement has to be decided by the development organization. As technology changes fast, the organization may release newer versions of the product and retire the previous versions. So, if a customer asks for a previous version of the software, the developer will not be able to supply it but may offer the new version. So, the developer has to give a backward compatibility to the software. Software products in the last few years have different life times. Many application software packages have a lifetime of few months, whereas Operating Systems like DOS (Disk Operating System) have outlived their expectations. So, even for the developer it is difficult to predict as to when the software is likely to retire. However, on a regular basis, the project manager has to review the necessary maintenance or support to be provided, the associated cost and return on investment. We can bring out a quality product when quality is ensured in every phase of development. Software Quality Assurance (SQA) is the process of ensuring quality in every phase of software development. It addresses both technical and managerial aspects of development.

Software Development Life Cycle and Quality Assurance

39

2.8 Life Cycle Models A software product is first conceived, developed and then delivered. After delivery, the software may undergo changes—the client may find some defects, which need to be rectified, or the client may ask some enhancements. So, after the development phase, there will be a maintenance phase till the software retires. This entire cycle is known as the life cycle of software development. The software development process is divided into a number of phases, each phase with a defined input and a defined output. However, it is not practical to use the same model for development of software for all types of projects. For instance, if the requirements are not clear, it will be a futile exercise to write the SRS document. For research-oriented projects, it is difficult to even have a precise problem definition. To facilitate software development for different types of projects, different life cycle models have been proposed. In this section, we discuss the most widely used life cycle models.

2.8.1 Waterfall Model The waterfall model is widely used for most of the commercial software development projects. This model is shown in Figure 2.9. System Requirements Specifications

System Design & Software Design

Implementation & Unit Testing

Integration & System Testing

Operation And Maintenance

Figure 2.9 Waterfall Model

Chapter 2

40 In this model, the software development process is divided into five stages:

y y y y y

Requirements specifications. System design and software design. Implementation and unit testing. Integration and system testing. Operation and maintenance.

Requirements Specifications In this phase, the project team will work out in detail, the functionality and the likely limitations of the software product to be developed. A document will be prepared clearly indicating the requirements. It is called the Software Requirements Specifications (SRS) document.

System and Software Design In this phase, from the requirements specifications, the system design is carried out. System design implies hardware design and software design. The functionality of hardware and software are separated out and design of the software modules is carried out. A test plan is also prepared describing the details of tests to be carried out on the system once development is completed.

Implementation and Unit Testing In this phase, the design is converted into code. The software is divided into modules and each module is further divided into components/units. A component/unit can be defined as a logically separable part of the program. Each component/unit is tested separately and ensured that it works without any defects.

Integration and System Testing In this phase, all the components/units are combined together and the system is built. The complete software is tested for its functionality and performance requirements. A test report containing the test results is prepared.

Operation and Maintenance In this phase, the system is delivered to the client and if any enhancements are required, they are carried out. This phase goes on till the software is retired.

Outputs of Different Stages in Waterfall Model The attractive feature of waterfall model is that at the end of each stage, a visible output is available. The outputs at the end of each stage are given in Table 2.1.

Software Development Life Cycle and Quality Assurance

41 Stage

Outputs

Requirements specifications

SRS document Draft user manual Maintenance plan

System design and software design

System design document Hardware design document Software design document Interface design document Unit test plan System test plan

Implementation and unit testing

Program code Unit test report

Integration and system testing

System test report Final user manual Working system

Operation and maintenance

$ (if the software is without any defects)

Table 2.1 Outputs at Different Stages in Waterfall Model

Advantages of Waterfall Model y Because of availability of output at each stage, it has high visibility. The project manager as well as the client will get a feel that there is considerable progress.

y Project monitoring, both by the internal management team and the client, is easy because of the visible outputs at each stage. During project planning, the manager and the client can set deadlines for each stage and check whether the progress is as per these milestones.

y As it is a conceptually simple model, it is followed in a large number of organizations, particularly for commercial projects.

Limitations/disadvantages of Waterfall Model y Freezing the specifications is extremely difficult. In such cases, a baseline approach is

followed. Baseline is defined as the certified output of one stage passed on to the next stage. The SRS document is written and even though there are some issues still to be resolved, the design is taken up. While doing the design, if any changes are to be made in the SRS, a formal procedure is followed for making the changes on the baseline document.

y For long-term projects (projects of duration one year and above), waterfall model requires freezing of the hardware. With fast developments in hardware technology, freezing the hardware may not be appropriate.

Chapter 2

42 y In cases where the client wants the developer to evolve specifications in a gradual manner, this model is not suitable.

A misconception about the waterfall model is that once the specifications document is written, this document cannot be modified. That is just impossible in any software project. You can always modify the specifications document; however, you need to follow a formal process of getting the modification approved by a competent authority before making the change.

In spite of these limitations, waterfall model is widely used in most commercial projects; as these projects are generally of short duration (anywhere between three months to one year) and for most of the projects, user requirements are very clear. Here are some projects for which waterfall model can be used for development:

y Development of a database management system for applications such as inventory management, library automation, banking software.

y E-commerce portal development. y Development of network protocol software. In waterfall model, the development is divided into five distinct phases: requirements specifications, system design and software design, implementation and unit testing, integration and system testing, and operation and maintenance. Waterfall model is the most widely used model for commercial projects as this model has well defined outputs at the end of every stage and hence monitoring the progress of the project will be very effective.

2.8.2 Prototyping Model For some projects, it is very difficult to obtain the exact user requirements (for example, when the user is not an IT-literate). In such cases, a prototype is built and demonstrated to the user; and based on the feedback, the SRS document is prepared. As shown in Figure 2.10, the prototyping model is used for finalizing the SRS.

Preliminary user requirements

Specification design implementation testing

Figure 2.10 Prototyping Model

Demonstration to client

SRS document

Software Development Life Cycle and Quality Assurance

43 For example, consider development of software for a real estate agent. The real estate agent wants to automate his activities by storing information about the available houses and the potential clients for renting the houses. As the real estate agent is unlikely to have a degree in computer science, you cannot ask him to give very detailed specifications. So, to start with, you will have a detailed discussion with the real estate agent to find out his broad requirements and then develop a prototype. You can demonstrate this prototype to the real estate agent and then obtain his feedback. When the agent sees the software, he will be able to give more details about his requirements and also tell what is missing in the prototype. Based on this feedback, you can write the clear specifications and then use the waterfall model to take up the software development. It is important to differentiate between prototyping as applied to hardware and as applied to software. In hardware, prototyping is used to validate the design. In software, prototyping is used to obtain the user requirements. Prototype is generally developed at the cost of the developer, and not at the cost of the client. So, prototyping has to be done with minimal resources. Prototype is developed using prototyping tools, such as scripting languages or Rapid Application Development (RAD) tools. When the prototype is demonstrated to the user, the user gets clarity of the functionality of the software and suggests required modifications. Prototyping is generally required to obtain user interface requirements. Based on the feedback, a more accurate SRS document can be prepared and the development work can start, using the waterfall model.

Advantages of Prototyping Model y For developing software for users who are not IT-literate, prototyping is a good approach.

y As SRS will be frozen after obtaining the feedback from the user, no changes are likely to be there in the requirements.

y When the client is not sure of the developer’s strength in software development, the client can ask the developer to build a small prototype, which can be used to judge the capabilities of the developer.

Disadvantages of Prototyping Model y Once the prototype has been developed and demonstrated to the user, the code

may not be of much use if it is written in a scripting language, which is not efficient. That is the reason, the prototype is called a ‘throw-away’ prototype.

y The development cost has to be borne by the developer. Generally, the client does not pay for the development of the prototype.

In spite of these limitations, prototyping is done extensively as it gives clarity of user requirements. If a prototype is not developed, then there is a possibility that there will be

Chapter 2

44 many modifications in the user requirements, which results in rewriting the software many times. Here are some projects for which the prototyping model is very useful:

y Development of workflow automation software for a small-scale enterprise. y Development of web services software for an online bookstore. y Development of an e-learning portal for a school. Prototyping model has to be used when the user requirements are not clear; or when the end user is not an IT-literate. If a prototype is developed and demonstrated to the user, the user will be able to come out with the requirements very easily. The cost of developing the prototype is generally borne by the development organization. Hence, the prototype has to be developed fast using prototyping tools such as Rapid Application Development (RAD) tools and scripting languages.

2.8.3 Evolutionary Development Model In this model, the system is built in stages. As shown in Figure 2.11, the initial user requirements are obtained and the product is developed. The user validates the system and gives the feedback for enhancements. The product is refined based on the new requirements. This process is repeated till a user-acceptable product is built. So, this model is also known as incremental building model.

User requirements

Specification design implementation testing

Validation of the system by the user

Is Y user satisfied?

Final product

N User

Figure 2.11 Evolutionary Development Model Suppose you are asked to develop an Internet portal for job seekers. Users can post their bio-data on the portal and also the employers can post their job openings. To start with, you may develop the portal with these minimal features. Later on, you may provide the following additional features, but one after the other: matching jobs with the bio-data, search capability for a specific type of job, on-line payment by the job-seekers through credit card, etc.

Software Development Life Cycle and Quality Assurance

45 Advantages of Evolutionary Development Model y This model is useful in exploratory programming (such as Artificial Intelligence applications) where it is difficult to frame the specifications.

y In case major problems are foreseen, the developer can stop the development after a few iterations.

This model is very appropriate for research projects. For example, to develop software for automatic speech recognition, a small vocabulary can be taken and the system is developed. After achieving success, the vocabulary can be increased in stages. This approach is better than starting development of an unlimited vocabulary speech recognition system directly (and after two years, realizing that it is very difficult!).

Disadvantages of Evolutionary Development Model y Because the project is open-ended, no time frame can be set. y Project monitoring is difficult. y Less visibility as compared to waterfall model. Evolutionary development model is very useful for research projects. Even for commercial projects, if there are too many features for the software, the client can prioritize the features and the most important features can be implemented in the first version. Later on, the features can be added incrementally. However, it is important to carry out the design keeping in view the future versions. Otherwise, there is a possibility that the present design will not be suitable for future enhancements. In evolutionary development model, also known as incremental-building model, the software is developed incrementally by adding features in stages. This model is suitable for high-end research projects.

2.8.4 Spiral Model In 1988, Boehm proposed this model. It is a very useful model for projects involving high risk. As shown in Figure 2.12, each loop in the spiral represents a development phase (any number of phases can be defined in a project).

Chapter 2

46 Objectives Alternatives Constraints

Plan the next phase

SRS Design Execution

Detailed design

Risk analysis Evaluation of alternatives

Figure 2.12 Spiral Model Each loop is split into four sections, each section to carry out a specific task:

y y y y

To determine the objectives, alternatives and constraints. Risk analysis and evaluation of alternatives. Execution of that phase of development. Planning the next phase.

For each round of the spiral, one form has to be filled containing the following information:

y y y y y y y y

Objectives: What are the objectives of this phase in terms of technical activities? Alternatives: What are the alternatives available to achieve the objectives? Constraints: What are the difficulties foreseen in the implementation? Risk factors: What are the risk items and the impact of the risk items? Risk resolution: How the risks can be resolved? Results: What results have been achieved vis-à-vis the objectives? Plans: What are the plans for the next phase? Commitment: What are the commitments to the client and whether they can be fulfilled based on the results of this phase of development?

So, at the end of each spiral (phase), the management can review the status based on this form, and decide the future course of action.

Software Development Life Cycle and Quality Assurance

47 Suppose you have to develop new product that involves hardware development and software development. It is a high-risk project as you are planning to come out with a product using the latest technologies and you are not sure whether you will be successful or not. So, there is a technology-risk. You can use the spiral model and divide the project into different phases such as SRS, design, detailed design, implementation and testing. The first phase (the first spiral) is the SRS. As mentioned above, for this phase, you plan your activities, carry out risk analysis and execute that phase. At the end of this phase, if you feel that you can go further, then you take up the design. If you feel that the risk is very high, you may consider dropping the project. Or, if you feel that the risks can be managed, you work out the details of how to overcome the risks. For instance, when you are developing a new product that involves hardware development, you may feel that there is a risk in meeting the timing constraints. In such a case, you may like to develop a prototype and ensure that the timing constraints can be met. So, you can add another spiral (phase) that is for prototype development.

Advantages of the Spiral Model y This is a flexible model, the phases can be determined by the project manager depending on the type and complexity of the project.

y This model is suitable for high-risk projects. y Project monitoring will be very effective as risk management is in-built into the

model. Periodically, the project can be assessed for its risks vis-à-vis the progress and a decision can be taken at the end of each spiral regarding its continuation or otherwise.

Disadvantages of the Spiral Model y This model is not suitable for low risk projects or projects without any risk. y It is a ‘complicated’ model for projects with clear SRS. Spiral model is a very flexible model that can be used for high-risk projects. The project is divided into a number of stages and at each stage, risk assessment is done to find out whether it is worth pursuing the project or not.

2.8.5 Synchronize-and-Stabilize Model For developing software products that are highly innovative and with a very short development time (Internet time), the above models cannot be applied. In such cases, the ‘synchronize-and-stabilize’ model, applied very effectively by Microsoft and Netscape [Cusumano 1999], would be very useful. This model is shown in Figure 2.13.

Chapter 2

48 Product vision statement

S Development sub-cycle & alpha release

P E C I F

Periodic system build

Development sub-cycle & beta release

I C A T

release

Development sub-cycle & final release

I O N S

Final product

Final Specifications and other work products

Figure 2.13 Synchronize-and-Stabilize Model To start with, a vision statement for the product is prepared which gives the broad goals of the product. For instance, “we have to develop a very easy to use operating system” is a vision statement for developing a desktop operating system. Based on this vision statement, and the inputs from market research, a document is prepared which lists out the important features to be incorporated, if necessary with prioritization—this document is a draft specifications document. The development teams work in parallel on various modules and periodically ‘synchronize’ the code by integration and testing. Initially, the synchronization is done less frequently and at later stages very frequently, sometimes daily. These periodic system builds are tested for usability, functionality and reliability; and feedback is given to the development teams. During the total planned development time, three or four milestones are defined—these milestones are used to ‘stabilize’ the product. Three typical milestones are alpha release, beta release and final release as shown in Figure 2.13. This flexibility in the development approach coupled with early integration and testing gives a good ‘feel’ of the product in initial stages itself. Also, the feature prioritization helps in releasing a product with the most important functionality and also helps in meeting the time target.

Software Development Life Cycle and Quality Assurance

49 The special feature of this model is that the specifications (and other work products) are complete only when the product is ready. However, after the alpha testing, the product can be demonstrated to the prospective customers and their feedback can be incorporated into the product. Also, from a management point of view, the model is attractive because the management can start getting a feel of the product from the beginning itself. Also, small teams work on different modules and integrate the modules periodically. This model has been used extensively by many innovative product development organizations. It is the best model for development of products using cutting edge technologies and with tight schedules (death march projects). Synchronize-and-stabilize model is used to develop highly innovative products. In this model, the broad specifications of the product are defined and development is taken up. As the development progresses, the specifications are also refined. The final specifications will be ready just when the product is about to be delivered. Microsoft Corporation used this model to develop its operating systems and Office suite products. Testing the software is given lot of thrust in this model. You need to keep on integrating and carry out system testing, preferably along with the potential customers, to give feedback on the software features.

Advantages of Synchronize-and-Stabilize Model y Because of the intermediate ‘releases’ of the product, the product can be made feature-rich by incorporating the necessary feedback of the prospective customers.

y The periodic system building approach paves way for testing the software for both functionality and performance.

y The integration problems encountered in large projects using other models are eliminated in this model, again because of the periodic system building.

y Project monitoring will be easy as there are intermediate milestones, such as alpha release and beta release.

Disadvantages of Synchronize-and-Stabilize Model y The detailed specifications document will be made available only at the time of product release.

y Periodic system builds require a rigorous process to be defined for integration of various modules.

y A parallel independent test team needs to be in place. However, for innovative product development, the disadvantages mentioned above can be ignored. To experiment with various alternatives and arrive at the right solution and get a good feel of the product from the beginning itself is important for making a

Chapter 2

50 product that would make an impact. Microsoft, one of the most innovative software development organizations, used this model for development of all its operating systems and Office products (such as MS Word, Power Point, Excel, etc.).

2.8.6 Summary of Development Life Cycle Models Table 2.2 gives a summary of the different models based on which a particular model can be selected for a specific project. Development life cycle model

Features

Waterfall model

It is used when the SRS is clear. It is a simple and effective model for commercial projects with clear requirements. Project monitoring is effective as defined outputs are available at the end of each stage.

Prototyping model

It is useful when the requirements are not clear. Prototyping generally at developer’s cost. It is a good model for projects for IT user organizations.

Evolutionary development model

It is useful for research projects. Incremental building helps reduce risk for commercial projects.

Spiral model

It is useful for high-risk projects. It is a complicated model for projects with clear specifications.

Synchronize-and-stabilize model

It is useful model for highly innovative product development and projects with strict deadlines. Specifications will be available only at the time of releasing the product. The periodic system builds give a feel of the product from the beginning to the management.

Table 2.2 Summary of Software Development Life Cycle Models The choice of a specific development life cycle model depends on many factors, such as: availability of clear user requirements at the beginning, whether the user is an IT-literate, whether there are major risks associated with the project, whether the product development involves high-end research etc. In all life cycle models, testing is an important phase, though what to test and when to test differ from model to model. But the golden rule is “test early”— as early as possible in the life cycle. The earlier the testing starts, the better will be the software quality.

Software Development Life Cycle and Quality Assurance

51

2.9 IEEE Standard for Developing Software Life Cycle Processes The IEEE Standard [IEEE 1991] gives a detailed set of activities, which need to be carried out for development and maintenance of the software. This is a comprehensive standard covering all the technical and managerial activities. This standard also describes how these activities can be mapped to specific examples of software life cycles. Depending on the need and type of project, software development process can be divided into eight phases or five phases, etc. For instance, the eight-phase cycle can consist of the following phases:

y y y y y y y y

Concept exploration Requirements Design Implementation Test Installation and checkout Operation and Maintenance Retirement

The software life cycle based on five phases can consist of the following phases:

y y y y y

Project initiation Concept development Definition and Design System development Installation and Operation

For developing organization-wide process standards, the IEEE standard can be used as the basis. To summarize, based on the type of the project, a development life cycle model needs to be chosen. Choosing the right type of the model is extremely important as the success of the project depends on this choice. For a project with no clear-cut requirements, if you choose waterfall model, the project will never take off. So, for the success of any software project, the first and the most important step is to choose the right development life cycle model. Software development process has to be divided into a number of phases or stages. However, there is no strict rule as to how many stages should be defined. Based on your needs of the project, you can divide into different stages. The IEEE standards suggest different models with different phases, which can be adapted to suit your needs.

Chapter 2

52

2.10 Software Metrics In every engineering discipline, measurement is an important activity. Just as we quantify weight, height, etc. and use units such as gram, centimeter, to represent various parameters; in software engineering, measures (or metrics) are used to quantify various parameters. The software metrics help in:

y Estimating the size and complexity of the project. y Tracking the progress of the project. y Analyzing the effectiveness of the software process and taking corrective action if necessary.

y Measuring the productivity of the people. In management, it is said that you cannot manage what you cannot measure. For an effective management of the software projects, the metrics are very important. Some important software metrics are described in this section.

2.10.1 The Person Month The effort required for execution of a project is measured in Person Months (PMs). If a person works for one month, the effort is one PM. If six people work on a project for one year, the effort is 72 PMs. Though the exact time a person works in a month may vary, on an average, 154 hours can be assumed. (22 days at the rate of seven hours per day, excluding one hour for social formalities, gossip, tea breaks, etc.) If the complexity of the project is estimated in terms of PMs, it is easy to calculate the cost in terms of dollars, if the rate per person hour is known (or assumed).

2.10.2 Product Metrics To estimate the size of a project is very difficult task. One accepted method of estimating the product size is using the metric KDSI (Kilo or Thousand Delivered Source Instructions) or Kilo Lines of Code (KLOC)—how many thousands of lines of code are required for the project excluding the comments. Based on the KDSI, a project can be categorized as small, intermediate, medium, large or very large: Small Intermediate Medium Large Very large

2 and 8 and 32 and 128 KDSI

In database management system (DBMS) projects, the project size can be measured in terms of number of tables, forms and reports.

Software Development Life Cycle and Quality Assurance

53

2.10.3 Productivity Metrics The number of lines that can be written by a programmer in one hour can be used as a metric for measuring the productivity of the programmer, i.e. the Delivered Source Instructions (DSI) per hour. Using the size of the project in KDSI and the average productivity of the programmer, the time required to execute a project can be calculated as follows: Time required for execution of a project (in hours) = Total KDSI of the project / (average KDSI per hour)

Another metric used for productivity is the number of defects removed per hour by a programmer. However, it needs to be mentioned that organizations should not judge the performance of individuals based on these metrics. These metrics must be used only to estimate the time required for the execution of the project or time required to remove the defects, etc.

2.10.4 Quality Metrics To measure the quality of the product, the following metrics can be used:

y y y y

Number of defects found per KDSI (known as defect density) Number of changes requested by the customer after the software is delivered. MTBF (Mean Time Between Failures) i.e. the average time between failures. MTTR (Mean Time to Repair) i.e. the average time required to remove a defect after it is detected.

To measure the quality of the output of a development phase, the SQA team has to define the metrics. For instance, during the requirements engineering phase, SRS document is written. Then design, implementation and testing are done. While doing the design or implementation or while carrying out the testing, the SRS document may have to be changed. The number of changes made to the SRS document can be a metric to measure the quality of the requirements engineering process. Similar measures can be worked out for the design, implementation and testing phases as well. The SQA team has to define and measure the software metrics at every stage of software development. The product quality metrics are different from process quality metrics. The product quality metrics reflect the quality of the product whereas the process quality metrics reflect how well the process is defined.

Chapter 2

54

2.11 The Management Processes In addition to the process definition described above, a number of management processes have to be defined so that quality product is delivered. These management processes include:

y Software Project Proposal Preparation: To give a proposal to the customer indicating the time frame and development cost.

y Software Project Planning: To prepare a detailed plan for executing the project, clearly indicating the timeframe and effort required for each phase.

y Recruitment: To recruit people with right skill sets for different aspects of development.

y Training: To train the team members on the required skills and application domain.

y Team Formation: To form the project team and ensure that there is a good coordination amongst the team members.

y Project Monitoring and Tracking: To continuously monitor the progress of the

project and take corrective actions through a defined review process. Reviews, such as SRS review, design review, code review, test results review, etc. have to be conducted in every phase of development. Quality has to be ensured in all management processes as well. Metrics have to be identified and measured for each of the management activities to study the effectiveness of the management processes.

2.12 Risk Management Risk is an uncertain outcome. If the probability of an occurrence of an undesired event is more, risk is more. Some undesired events cause more damage than others. So, a project manager has to identify what type of risks will be there in a software project. This is risk management. In this section, we will discuss about the risk management as applicable to software projects. As risk is an important concept to be dealt with even in testing process, a good understanding of risk management activities is very important. A well-defined risk management plan reduces the crises. To identify the possible risks for the project, it is important to obtain the perceptions and fears of different people [Gemmer 1997]. However, the general tendencies prevalent should be AVOIDED:

y y y y

Information is power, do not share it Managers do not like to hear bad news, share only good news No news is good news Take decisions based on emotions rather than business logic

Software Development Life Cycle and Quality Assurance

55 The manager has to create a culture wherein the team members tell openly the risks perceived by them—regarding the technology and time. Particularly the time risk is very high in many projects. The manager should not bulldoze his way through but listen carefully the apprehensions about the project risks. Through brainstorming discussions, the manager has to elicit the opinions of all the team members and identify the risks. Each risk has to be analyzed and the impact of each risk item on the cost, time and implementation has to be discussed and recorded. The necessary steps to overcome the risks are also planned. The various activities involved in risk management are shown in Figure 2.14. Risk assessment is carried out during the planning stage and risk control during the project execution stage. These activities are described in this section. Risk Management

Risk Assessment

Risk Identification

Risk Analysis

Risk Prioritization

Risk Management Planning

Risk Control

Risk Resolution

Risk Monitoring

Figure 2.14 Risk Management Activities Risk management involves the following activities: risk assessment and risk control. Risk assessment consists of risk identification, risk analysis and risk prioritization. Risk control consists of risk management planning, risk resolution and risk monitoring.

2.12.1 Risk Item Identification At the time of acceptance of a project, there will be many uncertainties, but still, we accept the project. These uncertainties are the root cause of risk. In addition, there will be internal risks, such as suddenly some key people leaving the project as a result of which the time frame cannot be met. Boehm identified the 10 top risk items based on a survey of a large number of software managers. These risk items are:

y Personnel shortfalls: It is the on-availability of skilled people for the job. y Unrealistic schedules and objectives: These time frames are over-committed and so are the features of the software.

Chapter 2

56 y Developing wrong software functions: Due to wrong interpretations or lack of full

understanding of the user requirements, software engineers develop wrong or unnecessary functionality into the software.

y Wrong user interface: User interface is the most complex part of the software design, where changes are likely if early feedback is not obtained.

y Gold plating (features which are marginally required): Creating flashy icons, adding frills to the software are a waste of time; the engineers do not agree, but managers know better.

y Continuous stream of requirements changes: After the initial discussions with the

client, you freeze the specifications. But the client will not. (“The software must be so flexible that making changes should be very easy for the next two years” or “specifications are a moving target” are the common remarks you hear from the clients).

y Shortfalls in externally furnished/purchased components: Interface with external

hardware or software is always risky. We assume things and start the project and realize that the assumptions are wrong. Enough thought has to be given to this aspect during the feasibility studies stage.

y Shortfalls in externally performed tasks: Think the case of interface with a legacy database. What appears to be an easy task at the outset will turn out to be a Herculean task.

y Real-time performance: For systems that need real-time response, such as process control systems, ensuring that the design would achieve the necessary performance is always risky. Unless the system is implemented, the response time cannot be measured and if the response time is not within the required limits, the whole project gets into trouble.

y Straining computer science capabilities: Some users want you to do certain things

that are not technically possible, with the state-of-the-art technology. Examples include: unlimited vocabulary speaker-independent speech recognition, natural sounding text to speech conversion, automatic language translation, etc. If the software developer has no knowledge of the application domain and accepts the project, then he is in trouble.

The risks associated with most of the projects are likely to be amongst these items. While carrying out the risk planning, the risks have to be identified with the above list serving as a good starting point. If there are any other risk items, keep adding to this list (a step towards better risk management). The top 10 risk items of a software project have been identified by Boehm after an exhaustive survey. It will be good to check whether any of these risk items are for your project on hand and take the necessary steps to avoid the likely damage of these risks.

Software Development Life Cycle and Quality Assurance

57 Identification of the possible risk items for the project on hand is the first and the most important activity in risk management planning. Some risks, such as manpower attrition, changing specifications are common to many projects but some risks, such as performance requirements, interface with external hardware/software are specific to some projects.

2.12.2 Risk Analysis For analyzing the risk, one can do the worst-case effort estimation. The impact of a risk item can be calculated by the loss. The loss consists of direct loss and indirect loss. The direct loss in terms of dollars is not difficult to estimate, but the indirect loss in terms of loss of credibility is difficult to estimate.

2.12.3 Risk Prioritization After identification of risks and assessing their impact, one can list the risk items in the order of impact or priority with which they need to be tackled. Of course, it is likely that the priorities may keep on changing. The prioritization helps in bringing the risk item into focus so that, the top management can address it immediately. Though there are a number of mathematical models to quantify the risk impact, in practice very few managers follow the models, and the impact is generally done based on subjective assessment. While doing subjective assessment, it is preferable if the opinions of other team members are also taken into consideration.

2.12.4 Risk Control Risk control is done during the review meetings. These meetings may be conducted exclusively to review the risk items and their impact or along with other project review meetings. A risk control team may consist of the project manager and other senior team members. Based on the priority and the impact, the senior management representative may also participate in the meetings. To control the risks, a risk management plan has to be prepared which describes the following:

y y y y y

What are the risk items? What is the impact of each risk item? Which risks have to be addressed with high priority? Who is responsible for risk management? What resources are required to overcome the risk?

For example, if important team members leave the organization jeopardizing the progress of the project, the risk becomes a high-priority risk and steps need to be taken to

Chapter 2

58 resolve the problem. The alternatives, such as recruiting a temporary consultant, subcontracting a portion of the work, informing the client of the problem and requesting for more time are the alternatives which need to be discussed and a decision arrived at. A good project manager is one who can foresee the risks, resolve the risks by continuous monitoring of the risk items and takes a corrective action before the risk leads to a crisis. The project manager should always strive not to get into crisis. Risk management has to be done using a process-oriented approach. A risk management plan has to be prepared and risk management has to be done as per this plan.

2.13 Documentation Many engineers dislike documentation work. However, the SQA team has to ensure that enough documentation is available at each stage of the project. The documents delivered at important milestones of a project are known as Work Products. Some important work products that need to be made available at various stages are:

y y y y y y y y y y y y

Project proposal Project agreement Project plan document Software requirements specification document Design document Configuration management plan Test plan Quality assurance plan User manual Source code with enough on-line comments Test reports Maintenance manual Documentation is a very important activity for all development and test engineers. That is the reason, written communication skills are very important. And, it is not at all difficult to gain expertise in documentation. After all, you need to write technical content, not poetry.

Software Development Life Cycle and Quality Assurance

59

2.14 Quality Management Systems When software development is sub-contracted to an organization, the prime contractor needs to ensure that the work is sub-contracted to a reliable organization, which can deliver quality software. So, the prime contractor has to ensure that the subcontractor has a defined quality system. To evaluate the effectiveness of the quality system, external auditing agencies will assess the quality system and certify it. The ISO 9000 standards and Capability Maturity Model Integration (CMMI) are two important Quality Management Systems that have been widely accepted in the software industry. SPICE (Software Process Improvement and Capability determination) is now another framework that is gaining popularity. The management of every software development organization has to decide which quality system has to be implemented. If the organization has already obtained ISO 9000 or CMMI certification, for each project, then the quality plan has to be in place based on these standards. All the quality standards have the same underlying philosophy—to develop the software based on well-defined processes, with the necessary documentation; and ensure that every employee is committed to the quality improvement program. The quality standards should be considered as facilitators to the quality program of the organization rather than a hindrance. The management of every software development organization has to create a culture whereby the team-members work within a well-defined framework to develop quality software by giving thrust to quality at each and every stage of the development process. The credit for the success of a project goes to the people who worked on the project and not to the processes. Similarly, the failure of a project is attributed to the people and not the processes. So, ultimately it is the people who are responsible for the success or failure of a project—and hence, the SQA team has to view the processes as facilitators for the development. Malcolm-Baldrige National Quality Award of National Institute of Standards and Technology (NIST, USA) is awarded annually for excellence for corporations, who provide leadership through best practices in management in various categories: manufacturing, service, small business, education and health care.

2.14.1 Quality Standards Quality is the mantra in industry. To deliver a product that is the delight of the customer should be the main objective of the project manager, every team member as well as the senior management. To achieve this objective, quality standards come in handy. The management of every organization has to adapt quality standards so that quality is ensured in every product that is delivered. Implementation of quality standards also

Chapter 2

60 gives visibility to the organization. In this section, we will study the internationally accepted quality standards for software development. Software engineering standards play an important role in defining the quality processes. There are nearly 250 software engineering standards developed by major standardization and professional bodes on various aspects—planning, testing, documentation, software metrics, Computer Aided Software Engineering (CASE) tool selection, etc. [Pfleeger 1994]. British Standards Institute defines standard as a technical specification or other document available to the public, drawn up with cooperation and consensus or general approval of all interests affected by it, based on the consolidated results of science, technology and experience, aimed at the promotion of optimum community benefits. In any standardization process, measures (or metrics) play an important role. However, it is very difficult, in the field of software engineering, to obtain objective measures, and eliminate subjectivity completely. The quantity denoted by one liter of oil is same throughout the world, but the number of lines of code in a program may be different when measured by different organizations. Hence, the software engineering standards should be treated as guidelines for improving the quality of the software rather than a ‘standard’ that will have universal applicability. There are international, national and organizational standards. Most of the organizational standards are derived from national or international standards. For software engineering, ISO 9000 and CMMI are the most widely used international standards, which are described briefly in the following sections. CMMI stands for Capability Maturity Model Integration.

2.15 ISO 9000 Series Standards The latest ISO 9000 series of standards of interest to software development organizations are:

y ISO 9000:2000 Quality Management Systems—Fundamentals and vocabulary y ISO 9001:2000 Quality Management Systems—Requirements y ISO 9004:2000 Quality Management System—Guidelines for Performance Improvement

The quality management system to be developed as per ISO 9000 guidelines is used for quality assurance when the supplier has to demonstrate the capability to the customer. The supplier and customer can be internal to the organization (for example, the development team and the marketing team of an organization).

Software Development Life Cycle and Quality Assurance

61 ISO9001:2000 standard gives the requirements to develop a quality management system. A summary of this standard with reference to software development is given below. The standard has eight clauses.

y Clause 1—Scope: The scope of the document is specified in this clause. This document defines the quality management system to deliver a quality product to the customer. This standard is applicable to all the organizations irrespective of the type, size or product/service being provided. ISO 9000 is applicable to a variety of organizations—hotels, hospitals, hardware development organizations, manufacturing organizations, and also software development organizations.

y Clause 2—Normative Reference: The references of this document are given in this clause. The important reference document is ISO 9000:2000, which describes the fundamentals and vocabulary.

y Clause 3—Terms and definitions: The terms and definitions as specified in ISO 9000:2000 are given in this clause.

y Clause 4.1—General Requirements: The management must have a well-defined

and documented quality management system that defines the processes to be followed to deliver a quality product or service. Using quantitative measures (metrics), the effectiveness of these processes has to be monitored and steps have to be taken for continual improvement of product and process quality. The processes should include both technical and managerial processes.

y Clause 4.2—Documentation Requirements 

Clause 4.2.1—QMS Documentation: management system includes:

The

documentation

of

quality

Š Quality policy Š Quality manual Š

Detailed definition of procedures mentioned in the quality manual

Š

Quality records The documentation need not be on paper (hard copy), it can be in computerized form (soft copy)..





Clause 4.2.2—Quality Manual: The quality manual should describe the processes to be followed for the entire software development life cycle. Alternatively, the quality manual can refer to other documents that give the details of the process definitions. Clause 4.2.3—Control of documents: As it is likely, that the documents related to the processes and projects will be changed periodically, it is necessary to establish a procedure for controlling the documents. Procedures have to be

Chapter 2

62 established so that there is clarity, on who is the approving authority for a document, who is the authority to modify the documents and who is the authority to distribute the copies and to whom the copies have to be distributed. It is also necessary to establish procedures, so that only the latest and relevant documents are made available. 

Clause 4.2.4—Control of records: Once the quality management system has been established, it is also necessary to maintain records to give evidence to the external auditing agencies that the quality system is being effectively operational. Documented procedures have to be established to maintain such records.

y Clause 5—Management Responsibility 





Clause 5.1—Management commitment: The top management must have a written quality policy indicating its commitment to developing quality products and/or providing quality service to the customer. The management has to define measurable quality objectives to meet the customer requirements and communicate the quality objectives to all the employees. Periodic reviews have to be carried out to ensure that the quality objectives are met. Clause 5.2—Customer focus: The ultimate goal of the management is to satisfy the customer. The management has to ensure that the customer’s requirements are understood well and after completion of the project, the customer is satisfied with the product. Clause 5.3—Quality Policy: Management’s commitment is reflected by its quality policy. When you visit an ISO 9000 certified company, the quality policy is displayed at prominent locations. A typical quality policy is as follows: ICS is committed to quality products and services for the total satisfaction of the customers. Through continual improvement of processes in all phases of product life cycle, ICS will strive to attain reputation to provide high quality products in the emerging areas of Information Technology. Everyone at ICS will adapt, nurture and uphold the quality processes of the organization. The quality of the policy has to be signed by the Chief Executive Officer (CEO) of the organization. Every employee of the organization should understand the spirit of the quality policy and commit himself/herself to the quality policy.



Clause 5.4—Planning Š Clause 5.4.1—Quality objectives: Based on the quality policy, the management has to define quality objectives, such as increasing productivity, reducing defects, etc. These objectives should be measurable. Š Clause 5.4.2 Quality Management System planning: To meet the quality objectives, the management has to plan the quality management system by defining the details of the processes in the software development life cycle.



Clause 5.5—Responsibility, Authority and Communication

Software Development Life Cycle and Quality Assurance

63 Š Clause 5.5.1—Responsibility and authority: In many organizations, responsibility and authority are not clearly defined as a result of which the quality suffers. For instance, the project team does not know who is the deciding authority when a customer asks for change in specifications. The quality management system mandates that a clear organization chart is drawn, indicating the reporting structure and the responsibilities and authorities of each individual. Š Clause 5.5.2—Management Representative: The management should appoint a Management Representative who has the responsibility and authority to implement the quality management system. He/she is the person who reports to the management on the effectiveness of the quality management system and also suggests necessary improvements. The management representative also interfaces with the external agencies such as external auditors. Š Clause 5.5.3 Internal communication: Lack of proper communication among the teams and between the team members and the management leads to project delays and poor quality software. To avoid such problems, a proper procedure for internal communication should be established so that all the employees are in unison. 

Clause 5.6—Management Review Š Clause 5.6.1—General: The management has to periodically (say, every three months) review the quality policy, quality objectives and the quality manual to discuss the effectiveness and possible improvements. Records of such reviews have to be maintained. Š Clause 5.6.2—Review inputs: For management review, the inputs have to be obtained from customers, external auditors and internal auditors. The inputs can also be in the form of suggestions from the employees. Š Clause 5.6.3—Review output: After the management review, the output will be action points in the areas of additional resources needed for effective quality management system implementation, process changes to be incorporated, improvements in the products etc.

y Clause 6—Resource Management 



Clause 6.1—Provision of resources: The management has to ensure that the resources necessary to implement the quality management system are in place. Clause 6.2—Human resources Š Clause 6.2.1—General: To deliver a quality product, the employees working on the project should have the competence. By selection of the right people and by imparting the necessary training, the management has to ensure that the people have the necessary competence.

Chapter 2

64 Š Clause 6.2.2—Competence, awareness and training: The management has to ensure that each and every employee has the necessary understating of the quality, development processes and the products. Training has to be imparted based on the requirements of each individual, and the training records have to be maintained. 



Clause 6.3—Infrastructure: The management has to provide the necessary hardware, software, testing tools and also supporting services such as communication facilities, Internet access, etc. so that the employees can carry out their responsibility to meet the quality requirements. Clause 6.4—Work environment: A congenial work environment is a must to deliver quality products and the management has to provide such a work environment.

y Clause 7—Product Realization 



Clause 7.1—Planning of product realization: The quality manual gives the overall processes to be followed in the organization. Based on the quality manual, for each product to be developed, a separate project plan has to be prepared. The project plan gives the details of the quality objectives to be met, processes to be followed, documents to be generated and the testing methodologies to be followed and the acceptance test criteria. Clause 7.2—Customer Related Processes Š Clause 7.2.1—Determination of requirements related to the product: The project team has to obtain the requirements from the customer and document them in the form of an SRS document. This document should give the details of functional and non-functional requirements. Š Clause 7.2.2—Review of requirements related to the product: The project team has to ensure that the requirements of the customer are well defined, documented and can be implemented. Many times, because of communication gaps, the requirements of the customer will be different from what is written in the SRS document. So, the requirements have to be thoroughly reviewed by the customer and the project team; so that there are no gaps between what the customer wants and what the developer understands. There should be a written agreement between the customer and the development organization on the requirements specifications. Š Clause 7.2.3—Customer communication: The project team should formalize the procedure for communication with the customer; so that changes in the requirements, complaints etc. are handled effectively.



Clause 7.3—Design and Development Š Clause 7.3.1—Design and development planning: The procedures for the design and development should be established which include defining interfaces, identifying inputs and outputs, review process, verification and validation of the design, and controlling the design changes. The sub-

Software Development Life Cycle and Quality Assurance

65 processes include development planning, design and implementation, testing and validation, and configuration management. Š Clause 7.3.2—Design and development inputs: Inputs required for design and development of the product should be identified. These include the SRS document, design documents of other similar projects, etc. Š Clause 7.3.3—Design and development outputs: The outputs of the design and development phase should include the product that meets the requirements, product acceptance criteria, and the user manuals. Š Clause 7.3.4—Design and development review: The project manager has to periodically review the design and development to ensure that work is progressing in the right direction and the end product will meet all the requirements as specified in the SRS. Any potential problems have to be identified and appropriate action has to be initiated. Records of the reviews have to be maintained. Š Clause 7.3.5—Design and development verification: The project manager has to carry out the necessary verification of the design and development so that each and every requirement of the SRS is met. A document called Acceptance Test Procedure (ATP) has to be prepared and it should be ensured that the software is tested as per the ATP document. Š Clause 7.3.6—Design and development validation: The project manager has to do a validation of the design and development to ensure that the project is being implemented as per the plan. Validation has to be completed before the product is delivered to the customer. Records of the validation results have to be maintained. Š Clause 7.3.7—Control of design and development changes: During the design and development, a number of changes are likely in the SRS document, design document, etc. that are already prepared. Whenever a change has to be made to any of the work products, the impact of the change has to be estimated and conscious decision has to be taken on implementing or discarding the change. Records of such changes are to be maintained. 

Clause 7.4—Purchasing Š Clause 7.4.1—Purchasing process: The products purchased must conform to the specified requirements. For ensuring this, there should be defined procedures for selection of suppliers and verification of purchased products. Š Clause 7.4.2—Purchasing information: For purchasing a product, the manager has to specify the product specifications, the quality management system requirements (for example, whether the supplier should have ISO certification) and any other requirements such as installation, warranty, maintenance, etc. Š Clause 7.4.3—Verification of purchased product: At the time of giving the purchasing information itself, the project manager has to specify the

Chapter 2

66 acceptance test criteria for the proposed product and when the supplier gives the product, the verification has to be done as per this procedure. Sometimes, you may need to purchase readily available software package, such as a financial accounting software, anti-virus software, a personal firewall software, etc. Such software is called Commercial Off The Shelf (COTS) software. When you purchase COTS software, the purchasing clause discussed above is useful. 

Clause 7.5—Production and Service Operations Š Clause 7.5.1—Control of production and service provision: Once the product is developed, it has to be delivered to the customers. If the product has to be delivered to multiple customers (as in the case of a generic product), proper planning has to be done to make multiple copies, installation, and later maintenance of the product. If any special equipment is required for production and maintenance, that also needs to be planned. Š Clause 7.5.2—Validation of processes for production and service provision: In cases when the verification is not possible unless the product is installed in the customer premises, a process has to be established to obtain the feedback on the performance of the product from the customer and how the defects can be removed. Š Clause 7.5.3—Identification and traceability: During all stages of production, delivery and installation, the product should be identified and traceable. For hardware products, a unique serial number is given. For multiple copies of the same software, a unique identification for each copy can also be given. Š Clause 7.5.4—Customer property: For execution of some projects, the customer may give some of his property to the developer. For instance, the customer may give the required hardware to develop a device driver. In such cases, the development organization has to take proper care of the customer’s property and if the property is lost or becomes unusable, it has to be reported to the customer. In some cases, such as in the case of software maintenance projects, the customer may give the source code to be maintained. The source code is the intellectual property of the customer and proper care has to be taken to ensure that the intellectual property rights of the customer are not violated. Š Clause 7.5.5—Preservation of the product: A process has to be defined for identification of the product, packaging, storage and protection.



Clause 7.6—Control of monitoring and measuring devices: To ensure that the product meets the requirements, test and measuring instruments may be used, such as oscilloscopes, protocol analyzers, etc. It is necessary to ensure that these instruments are working well. Necessary procedures have to be established for

Software Development Life Cycle and Quality Assurance

67 this such as periodic calibration of the test equipment, to identify when the next calibration is due, etc. If software test tools are used, it is necessary to ensure that these tools are functioning properly before using them to test the software.

y Clause 8—Measurement, Analysis and Improvement 



Clause 8.1—General: As discussed earlier, metrics play an important role in process improvement. Management has to define metrics for product quality and process quality and use these metrics to check on the process effectiveness and improvement. For instance, employee productivity is one metric. If the turnover per employee is rising steadily, this is a good indication that the organization is improving. Clause 8.2—Monitoring and Measurement Š Clause 8.2.1—Customer satisfaction: After all, customer satisfaction is the most important objective for any organization. The management has to define the metrics to measure the customer satisfaction. For instance, the number of complaints received from the customer on a product in a given period is one such metric. Š Clause 8.2.2—Internal audit: An internal audit team has to be formed which audits the complete quality management system for its proper implementation across the organization. Necessary training has to be provided to the internal audit team on the auditing process. The auditors should not audit their own work. Based on the internal audit, the reports have to be generated which indicate the non-conformity, and follow up action is taken to correct the non-conformities. These internal audits have to be held at regular intervals, say every three months. Internal auditors should undergo a training program on the auditing process. Š Clause 8.2.3—Monitoring and measurement of processes: Each process defined in the quality management system has to be monitored and its effectiveness is measured using metrics. For instance, the number of design changes is a metric to measure the design process. If the number is high, then the design process has to be refined to reduce the number of design changes. Š Clause 8.2.4—Monitoring and measurement of product: For each product, metrics have to be defined such as number of non-conformities etc. so that this information can be analyzed to study the effectiveness of the product’s processes.



Clause 8.3—Control of non-conforming product: If a product is found to be non-conforming to the requirements, appropriate action is to be taken to ensure that it is not delivered to the customer. If the non-conformity is known after delivery, it has to be reported to the customer. Records of non-conformities have to be maintained.

Chapter 2

68 



Clause 8.4—Analysis of data: The management has to collect data to study the effectiveness of the quality management system. This data can be from the customers, suppliers, process metrics, product metrics, etc. Clause 8.5—Improvement Š Clause 8.5.1—Continual improvement: The management should aim at continually improving the organization’s performance. To achieve this, the quality management system has to be continually improved based on the inputs such as customer feedback, external audit reports, internal audit reports, suppliers’ feedback, product metrics and process metrics. Quality policy, quality objectives and quality manual need to be regularly reviewed and improved to achieve overall performance improvement. Š Clause 8.5.2—Corrective action: When a product has non-conformities, the corrective action is taken to remove the defects. The cause for nonconformity has to be probed into and steps are taken to ensure that such causes do not recur. Records are maintained for the corrective actions taken. Š Clause 8.5.3—Preventive action: Potential non-conformities are identified and preventive action is taken. Records of corrective actions serve as good inputs to take preventive actions. Records of preventive actions taken are maintained.

Time Table for ISO 9000 Certification Software organizations that would like to obtain ISO 9001:2000 certification have to prepare a quality manual which gives the details of the development processes, prepare a quality plan for each project being executed, train all the employees on the processes, carry out internal audit and then make arrangements for auditing by the ISO 9000 certification bodies. A timetable for obtaining this certification is given below: Time

Certification

Month 1 to 3

Allocation of funds for quality certification All employees trained on quality processes Process documentation

Month 4

Hire certified quality auditor Audit existing process and recommendations

Months 5-6

Final quality manual preparation and audit the processes

Month 7

Send quality manual to certification body

Month 8

Review meetings on suggestions of the certification body Make changes as per recommendations

Month 9

Final on-site visit by the audit team and certification

This is only an indicative timetable. Organizations can certainly do a faster job.

Software Development Life Cycle and Quality Assurance

69

2.16 Capability Maturity Model Integration for Software Engineering (CMMI-SW) The vision of Watts Humphrey, Capability Maturity Model (CMM) was developed to assess the capabilities of organizations in taking up large software development projects of Department of Defense, USA. CMM addresses the process improvement in software development organizations. CMM identifies a set of guidelines that need to be implemented for producing quality software. The CMM framework has been accepted internationally as a very comprehensive framework for quality processes implementation. Software capability is defined as range of expected results that can be achieved by following a software process. The actual performance achieved by following these processes is known as Software Process Performance. The extent to which a process is explicitly defined, managed, measured, controlled and effective is defined as Software Process Maturity [SEI CMU 1994]. In organizations that develop both hardware and software, Total Quality Management (TQM) has to be followed. As shown in Figure 2.15, CMM addresses the management of quality of software aspects of the project, i.e. CMM is confined to software quality management of the organization whereas TQM addresses both hardware and software quality management. CMM

Project #2

Project #1 Hardware

Hardware

Software

TQM

Figure 2.15 TQM and CMM

Software

Chapter 2

70 A large number of organizations obtained the CMM certification. Subsequently, instead of CMM, CMMI is now becoming popular. Both CMM and CMMI follow the same philosophy of process improvement. As CMMI is the latest model, we will discuss this model in the next section.

2.16.1 CMMI Framework Software Engineering Institute (SEI) has been developing Capability Maturity Models, since 1991, for a number of disciplines, such as system engineering, software engineering, software acquisition, people management, etc. A number of international standards bodies such as ISO also developed models/standards for assessing the quality systems of organizations. Capability Maturity Model Integration (CMMI) for Software Engineering is the result of SEI’s efforts to develop an integrated improvement framework for the following standards for quality system management:

y CMM for Software (SW-CMM) version 2.0 draft C y Electronic Industries Alliance Interim Standard (EIA/IS) 731 y Integrated Product Development Capability Maturity Model (IPD-CMM) v0.98 CMMI framework is compatible with International Organization for Standardization/ International Electro-technical Commission (ISO/IEC) 15504 Technical report for software process assessment. In CMMI framework, there are two representations: ‘staged’ and ‘continuous’. An organization can choose either staged representation or continuous representation for its software process improvement. In staged representation, five maturity levels are defined and each level has specific process areas. An organization has to implement the process areas of level 2 and then graduate to level 3 and so on. If the organization chooses the continuous representation, it can select its own order of improvement. Staged representation is a proven sequence of improvements and an organization can migrate from CMM to CMMI easily. In this section, we will discuss the staged representation of CMMI for software engineering. The details can be found at [SEI CMU 2002].

2.16.2 Levels of Software Process Maturity Based on the software process maturity, an organization can be at one of the five maturity levels. The levels are shown in Figure 2.16. In CMMI framework, there are two representations: staged and continuous. In staged representation, five maturity levels are defined and each maturity level focuses on specific process areas.

Software Development Life Cycle and Quality Assurance

71 Level 5 Optimizing Level 4 Quantitatively Managed Level 3 Defined Level 2 Managed

Level 1 Initial

Figure 2.16 Levels of Software Process Maturity in CMMI The following characteristics describe the five maturity levels:

y Maturity level 1 (Initial): Organizations at this level execute projects using ad hoc

methods. Development is generally chaotic. It is likely that cost overruns and time overruns are common in such organizations. Fire fighting is a common phenomenon in such organizations because of lack of processes for executing the projects.

y Maturity level 2 (Managed): Organizations at this level follow a defined process for execution of each project. The requirements are managed as per a defined process. Every project is planned and executed as per that plan. Hence, a systematic procedure is followed to execute projects. Seven process areas are defined at this level and an organization is assessed as Level 2 organization if it implements all these seven process areas.

y Maturity level 3 (Defined): Organizations at this level have a set of process definitions across the organization and processes related to a particular project are derived from the organization-wide processes. As compared to level 2 organizations, processes are more clearly defined and efforts are made to continuously improve the process definitions. Processes are consistent across the organization. Quantitative measures are used to find the effectiveness of the processes. There are 11 process areas in this level.

y Maturity Level 4 (Quantitatively Managed): Organizations at this level define

quantitative objectives for process performance and product quality. Sub-processes are defined for the processes and wherever possible, these sub-processes are quantitatively managed. Predictability of process performance differentiates

Chapter 2

72 organizations at this level as compared to level 3 organizations. There are two process areas in this level.

y Maturity level 5 (Optimizing): Organizations at this level continuously improve

their process and product performance through innovation and technology. Organizations at this level address the common causes and find the root cause of the problems and improve the processes. Empowering the employees to bring in innovation is a must to achieve this objective. There are two process areas in this level.

The process areas for each of the maturity levels are given in Table 2.3. There are 22 Process Areas in total. Maturity Level

Process Areas

1: Initial 2: Managed

Requirements Management Project Planning Project Monitoring and Control Supplier Agreement and Management Measurement and Analysis Process and Product Quality Assurance Configuration Management

3: Defined

Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Integrated Project Management Risk Management Decision Analysis and Resolution

4: Quantitatively Managed

Organizational Process Performance Quantitative Project Management

5: Optimizing

Organizational Innovation and Deployment Causal Analysis and Resolution

Table 2.3 CMMI Maturity Levels and Process Areas Each of these process areas is described briefly in the following sub-sections.

Software Development Life Cycle and Quality Assurance

73 In CMMI framework, altogether 22 process areas are defined. Organizations need to implement process areas in level 2 and then move on to process areas in level 3. It is not advisable to skip the process areas at one level and go to the higher level.

2.16.3 Level 2 Process Areas Organizations at this level focus on managed processes. Each project is managed based on well-defined processes. There are seven process areas in this level.

Requirements Management This process area addresses the issue of obtaining the requirements from the customer. Based on the customer requirements, the product requirements and product component (such as operating system, database, etc.) requirements are decided. If there are any inconsistencies in the requirements, they are identified and resolved. Any changes to the requirements are also managed using a defined process. Commitment is obtained from the engineers that the requirements can be met. Inconsistencies in project plan and work products are also identified.

Project Planning This process area addresses the planning of the project—a very detailed project plan is prepared which contains all the details such as the estimates of time, effort and budget, identification of risk items and risk analysis, work products to be generated during the execution of the project, resources and training requirements, etc. The project plan document is periodically reviewed and updated.

Project Monitoring and Control This process area is to ensure that monitoring the project progress is done as per a defined process. Reviews are conducted to ensure that the plan is being followed and corrective action is taken if required. Risks, commitments and project plan are reviewed and monitored periodically and appropriate corrective actions are taken in a timely manner.

Supplier Agreement and Management When the product development involves purchase of a third party software such as testing tools, RDBMS, etc. a process is defined to identify the product components, select suppliers and evaluate the suppliers and for acceptance testing of the third party software. This process is followed to obtain the software components and integrate them into the product.

Chapter 2

74 Measurement and Analysis The management needs to be provided quantitative measures or metrics to monitor the effectiveness of a process or product. This process addresses this issue: identify metrics to be used, gather data and use the metrics for decision making and taking corrective actions. Initially, the metrics are done at project level and then at organization level.

Process and Product Quality Assurance This process area is to objectively evaluate the process and product quality. Noncompliance to the process or product quality is reported to the management and feedback is given to the development team. To objectively evaluate the quality, it is a general practice to have a separate quality assurance team though it is not mandatory.

Configuration Management This process addresses the issue of ensuring that changes made to various work products are done according to a specific procedure, i.e. to control changes.

2.16.4 Level 3 Process Areas In level 3 organizations, processes are defined in such a way that they are applicable across the organizations. Processes specific to a project are derived from the organization-wide process definitions. The processes at this level are more clearly defined as compared to those at level 2. There are 11 process areas in this level.

Requirements Development This process focuses on ensuring that the requirements of the software to be developed are captured, documented and validated. As a part of this process, the developer has to interact with the customer and obtain the customer requirements, identify the product requirements and product-component requirements and then validate the requirements. The product components are off-the-shelf software packages, such as RDBMS, test tools, etc. The customer may give the requirements in non-technical terms, which need to be converted into technical documentation.

Technical Solution Based on the requirements, the product is developed. The design, detailed design and implementation of the solution are the sub-processes of this process. To study the design alternatives, carry out a design and detailed design, obtain the off-the-shelf products if necessary, coding and unit testing, producing the necessary work products including user documentation form a part of this process.

Software Development Life Cycle and Quality Assurance

75 Product Integration The product is developed by integration of the different product components and by using formal testing methods, it is ensured that the product works correctly and delivered to the customer. Integration of the verified product components and validation of the product are done and the product is delivered to the customer along with all the necessary documentation.

Verification Verification is the process to ensure that the work products meet their specifications. This is done throughout the life cycle of the project. After the product development is complete, testing is done to verify whether the product meets the customer requirements, product requirements and product component requirements. Peer reviews play an important role in this process. Verification answers the question: “Are we building the product right?” If requirements are not met, corrective action is identified and implemented.

Validation Validation is the process to test whether the product meets the customer requirements in the intended environment. Validation answers the question: “Are we building the right product?” In validation, end users are involved.

Organizational Process Focus The objective of this process is to plan and implement an organization-wide process improvement program. Process performance objectives, such as productivity, defect removal rates, development time, etc. are defined and improvement areas are identified.

Organizational Process Definition The existing processes being followed in the organization are documented which can be used across the organization. These are known as process assets. An asset library is maintained which is a collection of all the process related documents, such as document formats, guidelines, checklists; life cycle models to be followed, etc. How the overall process definition can be tailored for different projects is also documented. A measurement repository is also maintained which gives the historical data such as estimates and actual values of project duration, effort, defects removed, etc.

Chapter 2

76 Organizational Training To ensure that the right people are assigned the right projects, training is of paramount importance. The management identifies the training needs of the people, trains them and also measures the effectiveness of the training program. The records of training are maintained.

Integrated Project Management Based on the process definitions of the organization, the processes required for the specific project on hand are identified. A defined process is followed for the project management based on the processes defined for the organization. Project planning is carried out based on the process assets and the measurement repository. The project also should contribute to the process assets of the organization. Proper interface to the customer is done to resolve issues.

Risk Management Risk management is an important process to prevent potential problems so that achieving the objectives is not jeopardized. It is a continuous activity throughout the project execution. A risk strategy is defined, risks are identified and analyzed, and a risk mitigation plan is implemented.

Decision Analysis and Resolution In the course of project execution, alternative solutions may be available, such as develop versus buy, which RDBMS to use, to take people on contract basis or on permanent basis, which life cycle development model to use, etc. This process is to analyze the alternatives based on certain criteria using an evaluation process to arrive at the decision. The subprocesses are as follows:

y y y y y

Evaluating alternatives, Establishing guidelines for decision analysis, Establishing evaluation criteria, Identifying alternative solutions Selecting a solution.

2.16.5 Level 4 Process Areas Level 4 process areas focus on managing the processes and product development using metrics. For implementing the processes defined at this level, the people involved should be trained in statistical analysis and should have access to statistical analysis packages. There are two process areas in this level.

Software Development Life Cycle and Quality Assurance

77 Organizational Process Performance The objective of this process is to quantify the effectiveness of various processes defined for the development of a software product. Metrics for process quality and product quality are defined and data for the earlier projects are collected. These can be used as the basis for predicting the expected results of the process performance for the future projects. As, it may not be possible to quantify the effectiveness of all the sub-processes, only relevant sub-processes are selected for measuring the process performance. For instance, consider software project effort estimation process. If historical data is available for a number of projects, based on this data, a model can be developed that can be used for estimation of the future projects. However, if this model was developed for say, database application software, you cannot use the model for embedded software development! So, the estimation process needs to be improved to take care of the application domain as well.

Quantitative Project Management At the start of a project, the project team will have some performance objectives— product performance objectives and process performance objectives. The aim of this process is to quantitatively manage the process of the project to achieve product/process performance objectives. The manager should be able to predict that the project will meet the performance objectives using statistical techniques, i.e. the project should be statistically managed. For example, the project manager should be able to predict the percentage of time required for design, coding and testing for the project. This again can be done if models are developed based on the data of the earlier projects. The project manager has to set performance objectives, use the statistical models to predict the performance for the present project and check whether these predictions are correct or not by actually measuring the performance. Since, it may not be possible to quantitatively manage all the processes, only some processes are selected for this activity. For effective management, quantifying the performance is of paramount importance. An important lesson in management is “you cannot manage what you cannot measure”.

2.16.6 Level 5 Process Areas Organizations at level 5 focus on continuous improvement through innovation in process improvement and technology. There are two process areas in this level.

Organizational Innovation and Deployment The goal of the management is not just to put a quality system in place and to follow it verbatim, but to continuously improve the processes so that the business objectives are met. This process area addresses this continuous improvement—to improve product quality, process quality, productivity and customer satisfaction using innovation and

Chapter 2

78 new technologies. To achieve this objective, the management should actively involve all the employees and even customers to obtain suggestions and inputs for improvement. The innovation can be in technical aspects or managerial aspects. Improvements can be in any of the phases of software development—planning, choosing the development life cycle models, using new technologies and tools, reuse strategies, new management techniques, etc. Each suggestion for improvement has to be analyzed for cost, effort, time; and cost-benefit analysis has to be carried out. A pilot study has to be carried out to ensure that the suggestion is likely to have a positive impact. The potential barriers to implement the suggestion also need to be studied—for instance, when a new technology is introduced, there may be opposition from some people who would resist change. Using objective measures, the impact of the suggestion has to be found out and if the suggestion is proved useful, it has to be implemented across the organization.

Causal Analysis and Resolution The essence of this process is learn from history. Based on the defects reported by the customers, quality assurance team members, or even the designers, the root cause of the defects has to be found out and recurrence of same or similar defects has to be prevented. This process helps in analyzing the cause of the defect and taking the preventive measures so that the same defect does not recur in future. After all, prevention is better than cure. The management of every organization has to strive to continuously improve its processes and technologies to maintain competitive edge. Organizations at level 5 will achieve this objective by continuous improvement, and learning from history is a very important aspect of improvement.

2.16.7 Importance of CMMI CMMI is the result of a combined effort of the industry, research and academic community. Most of the drawbacks of the SW-CMM are removed in this new model. This model also is compatible with other international standards and hence an organization, which is assessed for CMMI, can also get assessed for ISO 9000 and obtain the certification. CMMI is now finding wide acceptance with software development organizations across the world. Even if an organization does not have the resources to obtain the CMMI certification, the management can follow the guidelines of the framework to improve their process performance, product quality and employee productivity.

2.17 SPICE ISO/IEC International Standard 15504 ‘Software Process Improvement and Capability dEtermination’ (SPICE) is the latest addition to the models for software process

Software Development Life Cycle and Quality Assurance

79 improvement. This standard provides a framework based on which an organization can carry out:

y Process definition: To define the various processes needed to deliver quality software.

y Process assessment: Based on a set of attributes, assess how the process is implemented.

y Capability determination: To determine the capability of the process using a capability rating.

y Process improvement: To improve the process based on the present assessment levels.

The standard also specifies in detail, how to rate the processes, how to conduct assessments, how to construct, select and use assessment methods and tools as well as the qualifications and training required for assessors. In this standard, various processes required for software development--acquisition by the customer, supply by the vendor, design and development, installation and commissioning, maintenance support, management, resources and infrastructure and the reuse of software—are divided into nine process categories. These process categories are:

y y y y y y y y y

Acquisition Supply Engineering Operation Support Management Process Improvement Resource and Infrastructure Reuse

Each process is characterized by nine process attributes:

y y y y y y y y

Process performance Performance management Work product management Process definition Process deployment Process measurement Process control Process innovation

Chapter 2

80 y Process optimization Depending on how a particular process is implemented in the organization, the process capability maturity is indicated by six capability levels:

y y y y y y

0 : incomplete 1 : performed 2: managed 3: established 4: predictable 5: optimizing

During the process assessment, each process attribute is rated as follows:

y y y y

N: not achieved (0-15%) P: partially achieved (15-50%) L: largely achieved (51-85%) F: fully achieved (81-100%)

SPICE is now gaining popularity in a number of countries in Europe and in Australia. This model focuses on detailed assessment of an organization’s processes and hence one can get a good profile of the weaknesses and strengths of the organization by looking at the process dimension and process capability dimension. The important advantages of SPICE are:

y It gives an excellent framework for process assessment. y It defines the processes in more detail than CMMI or ISO 9001. y The assessment will be more objective because metrics are introduced even to grade the process attributes.

y An organization’s strengths and weaknesses will be evident by looking at the matrix that gives the process versus process capability.

y Some quality experts say that SPICE is suitable for small organizations also (with employee strength of even 10) whereas CMMI is more suited for large projects executed by large organizations. When an organization is audited for ISO 9001 certification, it will pass or fail. This is the main drawback of ISO 9001—there are just two levels 0 and 1. CMMI is a better because it has five maturity levels. SPICE, its proponents claim, will be much better because it assesses the organization’s process capabilities more objectively, and it can be implemented without less overheads.

Software Development Life Cycle and Quality Assurance

81

Summary y In the year 1968, in a NATO conference, the software crisis that prevailed in those years was addressed, and the term software engineering was coined.

y Software engineering addresses systematic development of software. y As per IEEE Standards [IEEE 1990], software engineering is the application of a systematic, disciplined, quantitative approach to the development, operation and maintenance of software, i.e. the application of engineering to software.

y Software engineering addresses the technical, managerial and administrative issues involved in developing quality software within the budget and within the time.

y In spite of the advances in the field of software engineering, the crisis of software development is still not overcome.

y A software project can be considered a success only when quality software is delivered within the time and within the budget and the execution goes smoothly without affecting the human relations.

y The quality triangle depicts the characteristics of a quality product. These characteristics are divided into: operational characteristics, revision characteristics and transition characteristics.

y Operational characteristics are: correctness, usability/learnability, integrity, efficiency, reliability, safety and security.

y Revision characteristics are: maintainability, testability, flexibility, extensibility, scalability and modularity.

y Transition characteristics are: portability, reusability and interoperability. y A process is defined as the step-by-step procedure to convert raw material into working product.

y Software development process is the detailed procedure to convert problem definition into working software product.

y Software development process is divided into a number of sub-processes or stages. These are: market analysis/research, feasibility studies, requirements engineering, design, construction or implementation, testing, maintenance and retirement.

y To produce quality software, quality has to be ensured in each and every phase of development. Software Quality Assurance encompasses all phases of development.

y Based on the type of project, different life cycle development models are used for software development.

y The most widely used software development life cycle models are: waterfall model, prototyping model, evolutionary synchronize-and-stabilize model.

development

model,

spiral

model

and

Chapter 2

82 y Waterfall model is the most widely used model for commercial software development. In this model, the development is divided into five stages: requirements specifications, system design and software design, implementation and unit testing, integration and system testing, and operation and maintenance.

y Prototyping model is used when the user requirements are not very clear and the user is not an IT-literate. In this model, a prototype is developed and demonstrated to the client/users. Based on the feedback of this demonstration, detailed requirements specifications are developed and then software development is taken up.

y In evolutionary development, the software product is developed incrementally. Features are added incrementally till the user is satisfied with the product. This model is suitable for research projects.

y Spiral model is used for high-risk software projects. Each spiral represents one phase of development. Any number of phases can be used depending on the complexity of the project.

y Synchronize-and-stabilize model is used for developing software products with lot of innovation. In this model, a product vision statement is prepared and the development is taken up. The software is integrated periodically and demonstrated to the users periodically. Hence, user feedback can be incorporated from the beginning itself.

y Based on factors such as availability of clear requirements, whether the user is ITliterate, whether risk is high, whether the project is a research project or a commercial project, etc. the development model has to be chosen.

y In software development and project management, metrics play an important role. Product metrics reflect the quality of the product, process metrics reflect the quality of the process.

y Documentation is an important development activity. The various documents generated at different stages of development are called work products.

y The three important standards based on which quality management systems are developed are: ISO 9000 series standards, CMMI and SPICE.

y Capability Maturity Model Integration (CMMI) framework has been developed at Software Engineering Institute, CMU.

y CMMI integrates the following models: CMM for Software (SW-CMM) version 2.0 draft C; Electronic Industries Alliance Interim Standard (EIA/IS) 731 and Integrated Product Development Capability Maturity Model (IPD-CMM) v0.98.

y CMMI

framework is compatible with International Organization for Standardization/International Electro-technical Commission (ISO/IEC) 15504 Technical report for software process assessment.

Software Development Life Cycle and Quality Assurance

83 y In CMMI framework, there are two representations: continuous and staged. In staged representations, five levels of maturity are defined.

y There are in total 22 process areas in CMMI framework. At each of the fivelevels,

process areas are defined. An organization that would like to obtain CMMI certification has to implement the process areas at level 2 and then move on to the other levels gradually without skipping the intermediate levels.

Review Questions 1. If software is developed without following any process, it results in: a) Lot of re-work b) Difficulty in maintenance c) Difficulty in keeping time and budget under control d) All of the above. 2. The term software engineering was coined in the year: a) 1971

b) 1968

c) 1994

d) 1990

3. Software engineering deals with: a) Coding issues

b) Design issues

c) Testing issues

d) All of the above

4. Software engineering principles need to be applied only in large development organizations. a) True

b) False

5. Usability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

6. Integrity is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

7. Efficiency is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

8. Reliability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

Chapter 2

84 9. Safety is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

10. Security is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

11. Maintainability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

12. Testability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

13. Flexibility is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

14. Scalability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

15. Extensibility is: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

16. Modularity is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

17. Portability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

18. Reusability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

19. Interoperability is a: a) Operational characteristic

b) Revision characteristic

c) Transition characteristic

d) None of the above

20. The budget required for a software project can be estimated very accurately a) True

b) False

Software Development Life Cycle and Quality Assurance

85 21. SQA encompasses all phases of development. a) True

b) False

22. You cannot take up design without finalizing the SRS totally. a) True

b) False

23. SRS document needs to be studied by the test engineers because: a) They need to find out whether the software can be designed b) They need to check whether the project can be completed within the time c) They need to check whether each requirement is testable or not d) They need to check whether the project is manageable. 24. Construction phase involves a) SRS finalization

b) design

c) coding/implementation

d) testing

25. The aim of coding should be: a) To reduce maintenance cost

b) To reduce testing cost

c) Both (a) and (b)

d) Neither (a) nor (b)

26. Which of the following is not a work product? a) Source code

b) Test plan

c) Design document

d) Operating system

27. Configuration management means: a) Managing the work sheets of the employees systematically b) Managing user requirements systematically c) Managing the project systematically d) Managing the changes in work products systematically. 28. Waterfall model is used when: a) User requirements are not clear b) Project is risky c) Requirements are very clear d) Innovative product development is involved. 29. Spiral model is used when: a) User requirements are not clear b) Project is risky c) Requirements are very clear d) Innovative product development is involved

Chapter 2

86 30. Prototyping is used when: a) user requirements are not clear b) project is risky c) requirements are very clear d) innovative product development is involved. 31. Synchronize-and-stabilize is used when: a) User requirements are not clear b) Project is risky c) Requirements are very clear d) Innovative product development is involved 32. Prototype has to be generally developed at: a) client’s cost

b) developer’s cost

33. In which model, a visible output is available at the end of each phase? a) Prototyping model b) Waterfall model c) Synchronize-and-stabilize model d) Evolutionary development model 34. In waterfall model, if SRS phase is over, the SRS document cannot be modified at all. a) True

b) False

35. Which model is used mostly in commercial projects? a) Prototyping model b) Waterfall model c) Synchronize-and-stabilize model d) Evolutionary development model 36. Microsoft used which model to develop its operating systems and Office suite? a) Prototyping model b) Waterfall model c) Synchronize-and-stabilize model d) Evolutionary development model. 37. In spiral model, you can define any number of phases of development as you like.. a) True

b) False

Software Development Life Cycle and Quality Assurance

87 38. Which model is useful for research projects? a) Waterfall model b) Prototyping model c) Evolutionary development model d) Synchronize-and-stabilize model 39. Which of the following metrics does not reflect the complexity of the project? a) Size of code in thousands of lines of code b) Effort in person months c) Number of people working and duration of the project d) Duration of the project. 40. If seven people work on a project for eight months, the effort in person months is: a) 64

b) 56

c) 8

d) 7

41. A programmer writes 12,800 lines of code in 200 days working eight hours per day. His productivity in lines of code per hour is: a) 10

b) 8

c) 128

d) None of the above

42. Which of the following is not a risk item? a) Non-availability of skilled people b) Changing requirements c) Completing the project before time d) Real-time performance 43. An ISO9001:2000 certified organization will always develop high quality products. a) True

b) False

44. CMM addresses: a) Software development industries b) Hardware development industries c) Hardware and software development industries d) All types of industries 45. The number of maturity levels defined in CMMI is: a) 4

b) 2

c) 5

d) 6

46. The number of process areas in CMMI is: a) 5

b) 11

c) 22

d) 10

Chapter 2

88 47. Risk management is a process area in CMMI. a) True

b) False

48. In SPICE, the process capability maturity is indicated by how many capability levels? a) 2

b) 8

c) 6

d) 7

49. Software quality metrics are used to: a) Measure the quality of the software process b) Measure the quality of the software product c) Measure the quality of the software process and software product d) Measure the performance of the programmers 50. Which of the following metrics does not reflect the reliability of the software a) Mean Time Between Failures b) Mean Time To Repair c) Mean Time To Fail d) Total effort in person months. 51. In a database application, response time is a: a) Functional requirement

b) Performance requirement

c) Scalability requirement

d) None of the above

52. Software quality assurance and testing are synonym. a) True

b) False

53. SQA is done to prevent errors. a) True

b) False

54. A CMMI level 5 organization will always produce defect-free software. a) True

b) False

55. ISO 9000 certification is awarded to only software development organizations. a) True

b) False

56. In CMM Level 1 organizations, software development is done systematically. a) True

b) False

Chapter

Fundamentals of Testing Chapter Objectives y y y y y y y y y

3

What is testing? Psychology of testing When to test and where to test? How to test? Test cases and test oracles Metrics in testing phase Criteria for completion of testing Risk-based testing Myths and realities of testing

In this chapter, the fundamentals of software testing are discussed by answering what, why, where, who, when, and how. We will also study the metrics to be used in testing phase. We will also answer a very difficult question in the testing phase: when is a testing complete?

3.1 What is Testing? Testing is a process to check whether the software meets the user requirements or not. During software development, the developers make many mistakes at different stages and testing helps in correcting those mistakes. So, in a broad sense, testing encompasses all phases of development—in every phase, the work products of that phase are tested. Hence, there is a testing activity corresponding to every phase of development. For example, in the requirements engineering stage, the SRS document is written and tested to check whether it captures all the user requirements or not. Though this is a very broad definition of testing, the various tests or checks carried out at different phases of development are more commonly referred to as Quality Assurance. Testing is generally considered as a separate phase after the implementation/ construction phase. Hence, once the code is converted into a working system, the testing phase starts. During the implementation or construction phase, the programmers write code. It is likely that the code will have a lot of errors. An error is a human action that causes the software to fail. The software is said to fail if the actual results do not match with the expected results. A flaw in a software unit/component that causes the failure is called a

Chapter 3

90 defect. Software engineers refer to a defect as a bug. Bug is a colloquial word; wheras defect is a scientific word. A flaw in a software work product that causes the software to fail is called a defect (commonly referred to as a bug). The human action that causes the software to fail is an error. Humans make mistakes, which produce faults or bugs or defects in the work products. When defective code is executed, it causes a failure of the system. The failures can be due to natural causes or man-made causes. If the hardware fails due to temperature variations, it is a natural cause. If lack of knowledge of programming language results in writing defective code, then it is a man-made failure.

The objective of testing is to find the defects. When the testing is done, the reasons for failures are identified and the defects are found. Then the defects are corrected. This is an iterative process—you need to test the software, find defects, correct the code, and test the software again. When you do the testing, your objective is to find defects, i.e. you look at the software with negative feelings and then do your job. This is called negative testing.

3.1.1 Testing versus Debugging Testing is different from debugging. Testing is done to find the defects, wheras debugging is done to find the cause of the defect. Debugging involves finding out which portion of the code is causing the defect, modify that code, and checking whether the defect has been removed. Debugging is the job of a developer. On the other hand, the test engineers need to find out the defects. They also need to be able to describe when the defect has occurred and had what happened after the occurrence of the defect—the preconditions and post-conditions. Suppose you are testing an application software package. If you give a report that the software ‘hangs’ suddenly, it is of no use. You need to report when exactly the software hangs (when I press the Submit button in the employee information entry form, after filling all the data). You may also need to specify the environment in which the software is tested (e.g. on a system with Windows 2000 operating system and Internet Explorer browser). Only when these details are given, the developer will be able to debug it.

3.1.2 Testing and Bebugging To check the effectiveness of the testing process, bugs are sometimes introduced intentionally in the software. This is known as defect seeding or bebugging. Suppose you have tested a software package and declared that it is OK. Now, the manager can intentionally introduce 10 defects in the code and you are asked to test it again. If you are able to find only nine defects, it indicates that you are unable to find

Fundamentals of Testing

91 10% of the defects that are introduced. Extending this logic further, it can be said that you are unable to find 10% of the total defects that were there in the original software. This technique of intentionally introducing defects is very useful to find out when the testing is complete: we will discuss more about this in a later section of this chapter.

3.1.3 Verification and Validation In the testing process, the two terms--verification and validation--have to be differentiated. Boehm defines these terms based on the answer to the following questions:

y Validation: Are we building the right product? y Verification: Are we building the product right? Validation is to check whether the software meets the customer expectations. This is done by testing the software for its functionality and other requirements as given in the requirements document. Generally, validation is carried out with the involvement of the customer. Verification is to check whether the software conforms to specifications. Verification is done by the development team at various phases of development. The SRS document, the design document, and the code are reviewed to ensure that the product is being developed using a process-oriented approach. Verification is an in-house activity of the development organization. Two other related terms are quality control and quality assurance. Quality control is defect detection or testing of a product. Quality Assurance is defect prevention of a product. In other words, quality control is validation and quality assurance is verification

Two other related terms are quality control and quality assurance. Quality control is defect detection or testing of a product. Quality Assurance is defect prevention of a product. In other words, quality control is validation and quality assurance is verification.

3.1.4 Root Cause Analysis While doing testing and debugging, it is important to identify the bug and remove it. More important is to find out why the bug has occurred in the first place, i.e. to find out the root of the cause of the problem. This is called root cause analysis. Root cause analysis is a very important aspect of testing and debugging.

Chapter 3

92 Consider the following example; where A program is written for controlling the temperature of an air conditioner. In the code, a variable ‘temp’ is assigned a value of 25 using the statement: int temp = 25;

When you carry out the testing, you realize that you cannot reduce the temperature of the air conditioner below 25. You find that the error is due to the above line of code; when you change the value to 16, the program works well. Certainly, you have debugged well. But what is the root cause? Is it the mistake that you made while coding? Or, is it a problem in the specifications? You can refer to the SRS document and check whether any value is mentioned for the minimum temperature that can be set. If it is not mentioned, or if it mentioned erroneously as 25, then the root cause is due to poor specifications and you need to improve the specifications process. In this case, the root cause is poor specification and the effect is writing a wrong code. Root cause analysis is given thrust in both ISO 9000 quality management system and CMMI framework. Root cause analysis helps in improving the total software development process.

3.2 Significance of Testing As human beings we all make mistakes, and these mistakes are reflected as defects in the software product. These defects make the software fail which are due to our limitations as human beings—ignorance, arrogance, lack of communication, politics within the team, jealousy, too much greed to save more money and so on. In spite of all our mistakes, we still need to deliver quality software. Hence, we need to test the software. Here are some reasons why testing the software thoroughly is a must:

y As a result of communication gap between users and requirements analysts, the software would not have been developed as per the user requirements.

y Due to the communication gap amongst developers, when a large number of

developers work on the project, the complete functionality required would not have been implemented.

y Individual programmers’ inability to do efficient coding would have resulted in defective code.

y Software has been tested on a simulated environment as the actual environment is

not readily available at the developer’s premises. Testing is necessary in the actual user environment.

y Software that goes into the embedded systems is generally developed on a PC or at workstation. It must be tested on the actual hardware before it is delivered to the customer.

Fundamentals of Testing

93 y Before the software is delivered to the customer, it must be ensured that privacy, safety and security of the users and user data are taken care by a rigorous testing— otherwise, the customer gets into problems and in turn will put the developer in trouble.

y Due to constraints of time imposed by the client, the software was developed in a hurry, and hence quality would have been compromised.

y Constraints of budget imposed by the client and then the management would have resulted in compromising on quality.

Due to various factors, the end product would not have been of good quality. But, it is the responsibility of the development team to deliver quality software and hence the need for testing. Quality assurance and testing have only one objective: to satisfy the customer. Ultimately, the customer satisfaction with the software has to be ensured and this can be achieved only through rigorous testing. .

3.2.1 Cost of Quality The ultimate objective is to develop quality software. But quality does not come free— you have to pay a price for it. You have to put in the required effort to produce quality product. This effort is the ‘cost of quality’. This cost has three components [Humphrey 1997]

y Failure cost y Appraisal cost y Prevention cost Failure cost is the cost of fixing the product defects. Appraisal cost is the cost of assessing the product to check whether it has defects or not (the cost of removing the defects is not included). We can look at this cost as the insurance paid to assure that the product is defect–free. Prevention cost is the cost of modifying the process to avoid defects in the software. For example, you may like to develop a prototype to reduce the defects due to improper specifications. Our aim should be to reduce the failure cost to the maximum extent possible, even if there is an increase in the appraisal cost and prevention cost. If the product is delivered to the customer and then the defects are found, the cost of repair is much more. So, the aim should be to reduce the repair cost to the maximum possible extent.

Chapter 3

94 Even during development, the cost of repair is high if the defects are found at the testing phase. So, the aim should be to reduce the defects well before the coding stage by a thorough review of the SRS document, design document, etc. Cost of quality has three components: failure cost, appraisal cost and prevention cost. The objective of the development team is to reduce the failure cost to the maximum possible extent.

3.3 Psychology of Testing Now, let us find the answer to the fundamental question: What is testing? Consider the following definition for testing: Testing is the process to prove that the software works correctly. If the person who has developed the software carries out the testing, he/she will try to use this definition. This is the typical psychology of testing. The person who developed the software will only try to show that the software works correctly. So, he will give only those inputs for which correct results are obtained, or press only those keys by which the software does not crash. Recall your college days, when the professor came in to check your program in the laboratory examination on data structures. If the professor tries to take the keyboard to test your program, you will not allow him to touch the keyboard! It is because, as the developer of the program you know for which inputs your program will work and for which inputs your program will not work. So, you only try to give the inputs to prove that your program works correctly. This is not just the student psychology, even in the industry; the developers keep doing the same thing. However, in a commercial environment, using this definition for testing is dangerous, you cannot deliver software, full of bugs, to the customer. But, the important point is that the developers always try to follow this definition of testing. From a developer’s point of view, the definition of testing is: “testing is the process to prove that the software works correctly”.

Now, consider the following definition: Testing is the process to prove that the software does not work This appears to be a very sadistic definition. Surprisingly, when the software is given to the QA department of your organization, they follow this definition. The test engineers’ joy knows no bounds whenever they point out that there is a bug! Strictly if the aim of the test engineer is to prove that the software does not work, then it can be considered a good testing process. So, if persons other than developers test the software,

Fundamentals of Testing

95 then they follow this definition. This type of psychology towards testing would bring out most of the defects. From a quality assurance/test engineer’s point of view, the definition of testing is: Testing is the process to prove that the software does not work.

However, there is a practical difficulty in this definition. How long you will go on testing after which you can confidently say, “Yes, the software works”? Many operating systems and application software packages which we bought have bugs but still, they have been released into the market, and millions of copies have been sold. You can never say that the testing is complete and that the software is defect-free—never! So, a realistic definition of testing is: Testing is the process to detect the defects and minimize the risk associated with the residual defects. This definition sounds very realistic because at some point of time, we need to stop testing and release the product to the client. We know very well that there are likely to be some defects in the software (residual defects or latent defects), but the risk of not rectifying them before releasing to the customer is not very high. So, this is a very practical definition that is used by the managers. From a manager’s point of view, the definition of testing is: “testing is the process to detect the defects and minimize the risk associated with the residual defects”.

To summarize, the psychology of the people also plays a role in testing the software. And, who is competent to test the software? Developer or test engineer? Testing the software by the developer has many advantages:

y Developer is likely to be very familiar with the user requirements, specifications

documents and the design. Hence, developer will have the necessary background and knowledge to generate the necessary tests.

y Developer knows the intricacies of the code written by him/her. Hence, carrying out code-level testing is easy for the developer.

y In case there are defects, debugging has to be done by the developer. Hence, testing should be done by the developer himself.

However, because of the mindset of the developers and psychology of testing, unfortunately, testing by developers does not yield good results. So, a separate testing team is formed in most of the organizations. A separate testing team will have the following advantages:

y The testing is done objectively. Hence, it is likely that many defects are detected. y Test engineers are specialized in testing and using testing tools required for their

job; so, they can do the job productively. For example, security testing experts are the persons trained to test the security aspects of software. If banking software has to

Chapter 3

96 be tested, it is better if the security experts test it and certify it, rather than just the developers testing it. When a separate test team is formed, it is very important that at least some test engineers be involved in the project from the beginning—preferably from specifications stage itself—so that the test engineer can check whether each and every specification is testable or not. The test engineers and developers need to interact regularly. To start with, the developers need to brief the test engineers on the software to be tested—its functionality, design aspects, etc. The test engineers need to periodically communicate the test results to the developers. This communication has to be very cordial. Only persons with good communication skills and inter-personal skills can survive as test engineers. The test engineers need to tell the developers that their software has problems, but in a very polished way.

In spite of the best efforts by the project managers, the development team and the testing team fight with each other. This is because, the developers are emotionally attached to their work and if someone finds out errors/defects, they feel hurt. On the other hand, the test engineers think that they are objectively doing their work to find the defects. It is important for everyone to realize that:

y The focus is to develop quality software; and only through team work the objective can be achieved; and, both developers and testers have a role to play.

y The test engineers need to communicate their results without malice towards the persons whose code has the defects.

y The developers need to appreciate that code written by anyone can have defects (recall those great fiascos we discussed in chapter 1).

If the definition: Testing is the process to prove that the software does not work is followed by the test engineers, then they can uncover many errors in the software and make the software reliable. However, a realistic definition of testing is: Testing is the process to detect defects and reduce the risk associated with the residual defects Quality is viewed differently by the customer, developer and tester. The different definitions result in what is known as quality gap.

3.4 Testing Choices During the last twenty years, the cost of hardware is decreasing drastically, thanks to the developments in semiconductor industry. At the same time, the cost of software is

Fundamentals of Testing

97 increasing very rapidly—due to higher manpower costs and infrastructure costs. As a result, many organizations, particularly those in the developed countries are outsourcing their complete software development activities. But they used to handle software testing and quality assurance activities themselves. Even that trend is changing now—many organizations are outsourcing even their testing activity. Consider a bank that gets its banking software developed by a software development organization in India. After the software is developed, it will be given to another organization, located perhaps in Malaysia, that focuses on testing the banking software— for the functionality, performance in a web-based environment, reliability, security, accessibility, internationalization, etc. Many IT user organizations are giving thrust to this third party testing. Even software development organizations are getting the software tested by independent agencies to ensure quality products. The managers of IT user organizations as well as software development organizations have two choices: to do the testing in-house or to outsource the testing activity.

3.4.1 In-House Testing In-house testing is certainly a good choice, if the necessary infrastructure and manpower are available. The advantages are:

y The intellectual property will not be given out to a third party. y The management will have total control over time and budget. y The in-house testing team has access to the developers and hence, the test engineers can interact with the developers on need basis.

On the down side, in-house testing has some difficulties too:

y Maintaining the infrastructure is very costly. y The necessary expertise may not be always available (for example, testing the security aspects of a banking software).

y Getting the manpower, training them and keeping them for the duration of the project is a challenge.

An organization has to consider the above aspects before deciding on in-house testing or outsourcing. The management will have full control over the activity if the testing is done in-house. However, the infrastructure and manpower costs will be very high.

Chapter 3

98

3.4.2 Outsourcing As testing has become very specialized, now a number of organizations are outsourcing their testing activity. This has many advantages:

y Software testing needs lot of infrastructure—high-end servers, network with high

bandwidth, testing tools, and specialized test instruments. The prime contractor need not maintain this infrastructure.

y The subcontractor will have the necessary human resources readily available— professionals trained in specialized areas, such as security testing, conformance testing to check whether the software meets international/national standards, etc. The prime contractor need not bother about recruiting and training the manpower.

y As testing professionals are experts in their area, their productivity will be very high and hence testing is done faster and better at the subcontractor’s premises.

y The problems associated with psychology of testing are eliminated as the test engineers belong to another organization.

y If the testing activity is outsourced to an organization in a developing country, then it results in lot of cost saving.

On the downside, the management will encounter a number of challenges:

y The intellectual property (such as: the SRS document, the design, even the source code) may be transferred to a third party. Ensuring that the third party will honor the intellectual property rights is a big issue to be tackled.

y The time frame is not under strict control of the prime contractor. y Evaluating the vendor, particularly for the first time is very difficult. Many prime

contractors feel that outsourcing creates problems—yes, when the choice of the vendor is incorrect, and then outsourcing will result in a bad experience.

y If the outsourcing organization is located in a different country, the laws related to intellectual property, copyrights and cyber-laws may be different.

Keeping in view, the pros and cons of in-house testing and outsourcing, the management has to decide very carefully where the testing has to be carried out. Outsourcing of testing activity results in cost savings. However, choosing the outsourcing organization, intellectual property protection and control over the time frame are the challenges.

3.5 Who does the Testing? As discussed in the section on psychology of testing, the development team tries to prove that the software works correctly. To overcome the difficulty, in many organizations, the development team will be different from the testing team. This would ensure that the

Fundamentals of Testing

99 testing is done without any bias and maximum possible defects are found before the release of the software. Because of the complexity of the testing, the test team must have sufficient manpower. In Microsoft Corporation, the ratio of development engineers to test engineers is 1:1. In NASA Goddard Space Flight Center, the ratio is 1:7, i.e. for every development engineer, there will be seven test engineers. Hence, depending on the complexity of the project and criticality of the application, test team size will be different. Large organizations can afford to have dedicated test teams for each project. However, in small organizations, it is very difficult to have such dedicated teams. Even then, the project manager has to make the software tested by people other than the developers. One possibility is to make another development team to test the software. Alternatively, test engineers can be recruited on a part time basis or on a limited time contract. The test reports generated by the test team have to be looked into impartially by the project manager—the aim of the project manager is to ensure that the quality product is delivered and not to point the finger at the person whose module did malfunction.

3.5.1 Developers as Testers If developers carry out the testing, there are some advantages:

y The developers are familiar with the user requirements from the beginning of the coding activity. Hence, they are in a better position to generate good use-cases.

y The developers are familiar with the structure of the program—both code as well as the structure of modules, sub-systems, etc. So, they can do the testing better.

However, they also know the limitations/defects of their software and hence, only developers doing the testing is not advisable. However, the developers have to carry out the following types of testing:

y Unit/component testing: Unit or component is the smallest piece of software that

can be tested independently. Only the developer can do the testing of the unit/component because he/she knows the details of the unit/component.

y Integration testing: While carrying out the integration of the different modules, the

interface details between modules need to be studied. This again involves the codelevel details. As test engineers are not familiar with the code, they cannot handle this activity. Hence, integration testing has to be done by the developers. Developers are in a better position to do the testing because they are familiar with the code and the functionality. Their involvement is a must for unit/component testing and integration testing. However, because of psychology of testing, system testing is done by an independent testing team.

Chapter 3

100

3.5.2 Independent Testing Team If an independent testing team does the testing, they can do an impartial testing without any bias. So, the software can be tested thoroughly. However, as test engineers are not the developers, there are a few issues to be tackled:

y The test engineers may not be in the project since the beginning of the project and

hence, they may not be familiar with the user requirements. So, the test engineers need to have an excellent knowledge of the application domain and also study all the work products—SRS document, design document, etc.

y If the test engineers have to do the code reviews also, then they need to become familiar with the programming language and the source code.

y The test engineers need to constantly interact with the developers to come out with a good testing strategy.

y The test engineers need to realize that the developers are emotionally attached to

their work (the code) and hence the test engineers need to be very careful when pointing out software bugs. You need to tell that the damn thing does not work, but tell that politely!

Test engineers can do the following testing with minimal interaction with the developers:

y System testing: System testing involves testing the software against the specifications. If the SRS document is available, the test engineers can study it and then the system testing can be done.

y Acceptance testing: Acceptance testing is done by the client in association with the

development organization. Even before the software is offered for testing to the customer, the testing team can carry out the testing and ensure that the software offered to the customer for testing is of high quality. The test engineers can objectively do the system testing by studying the work products, such as SRS document, design document, etc. The test engineers need to communicate test results in such a way that the sentiments of the developers are not hurt.

3.5.3 Buddy Testing When two people sit together and do the testing, it is called buddy testing. If a developer and a tester together work at unit level testing, and the test code is written while the actual code is being developed, then it leads to better software. However, it is difficult to implement this method because the tester also needs to get familiar with the code, and the resource (manpower) requirement is rather high. Note that buddy testing is a concept similar to eXtreme Programming (XP) concept wherein engineers work in pairs to do the development. Proponents of XP and buddy testing claim good results but these concepts are yet to be widely accepted.

Fundamentals of Testing

101 Now it has become a norm in the industry that the testing of the software has to be done by persons other than the developers. In organizations, which give thrust to testing, there will be a separate QA/testing team to test the software. All the work products are subjected to testing. In many organizations, test engineers’ job is only to carry out the testing after the development is completed. In some organizations, test engineers take on more responsibility of checking the various work products, such as SRS document, source code, user manual, etc.

3.6 Testing Phases If you start testing only after the software is developed completely, it is called reactive testing. On the other hand, if you start designing the tests in the early phases of development, it is called preventive testing. And, obviously, prevention is better than cure. Testing needs to be done on every work product that is produced during the development. Work products, such as SRS document and design document, are tested through reviews and inspections. Testing the software is done through dynamic testing by executing the code. During the course of construction/implementation, the testing is done at different levels such as unit level, module level, sub-system level and system level. Generally the testing up to the system level is carried out by the developers themselves. And, System testing is done by the testing team.

The V-Model depicts the testing to be done at various phases of development.

3.6.1 V Model To bring out a quality product, we need to test the output of every phase of software development. This is fundamental to produce quality software. This is reflected in the V Model. This model shows the various development phases and the corresponding testing phases.

Chapter 3

102 Operational/ Business Needs

Testing in the actual environment

User Requirements

Acceptance Testing

System Testing

SRS

Design

Integration testing

Component/Unit Testing

Figure 3.1 V Model In the software development process, the end user business/operational needs are converted into user requirements. From these user requirements, software requirements specifications document is prepared. Based on the SRS document, the design is carried out and the design document is made. Based on the design, the various modules are designed and each module is further divided into units/components. These activities are shown in the left side of the V Model. Each unit/component is coded and tested. The components are integrated into modules and these modules are tested. Module testing is done to check whether the design is correct. After the modules are integrated, system testing is done. System testing is to ensure that the software is as per the SRS document. Subsequently, the software is tested by the customer and this testing is called acceptance testing. Acceptance testing is done to check whether the software meets the user requirements. After that, the software is installed in the operational environment and tested. This is called operational testing. This testing is done to check whether the software meets all the business/operational needs of the user. The various levels of testing are shown on the right side of the V Model. The V Model indicates the various levels at which the software has to be tested, with the ultimate objective of developing software that meets the business needs of the customer.

Fundamentals of Testing

103 The V Model depicts the various phases of development and the corresponding testing activities at different stages.

Testing is involved at every stage of coding. In component testing, test cases can be prepared even before starting the coding of the component. This approach is called test first approach or test-driven development.

3.6.2 Testing and Life Cycle Models In Chapter 2, we discussed about various software development life cycle models. For commercial projects, the waterfall model is used. For projects in which the user requirements are not very clear, prototyping model is used. For research projects, the Evolutionary development model is used for projects involving lot of risk; Spiral model is used, for projects involving innovation and strict deadlines; Synchronize-and-stabilize model is used. In all these models, testing is certainly an important phase. But when to test differs slightly.

y In Waterfall model, testing phase starts after the module integration is done and

hence testing is done on a reasonably mature product. The testing is done using the SRS document as the reference.

y In Prototyping model, the prototype is tested based on the preliminary user

requirements obtained. Hence, at this level, testing is very rudimentary. However, after the user feedback on the prototype is obtained and the SRS document is finalized, the software is developed using the Waterfall model and testing is done on a mature product.

y In Evolutionary development model, the software is developed incrementally and hence testing is also done incrementally. This involves lot of testing effort because every time the system is incremented, the testing is done differently.

y Projects that use Spiral model are high-risk projects. Hence, in every spiral while studying the risk factors, the testability of the software has to be given lot of thrust.

y Projects that use Synchronize-and-stabilize model require very rigorous testing.

The testing team has to be in place from the beginning. As periodic builds are involved, testing has to be done thoroughly for each build and feedback given to the developers for further improvements. The testing rigor depends to a great extent on the chosen development life cycle model. Synchronize-and-stabilize model calls for a testing team to be in place from the beginning of the project.

Chapter 3

104

3.7 Testing the System Once the implementation stage is over, the software has to be tested. During the development itself, the unit/component testing and integration testing are done. Then, the system has to be tested. The system has to be tested with SRS document as the reference. The SRS document contains mainly two sections: One section contains the functional requirements and the other section contains the non-functional requirements. Functional requirements specify what the software does. This functionality obviously differs from application to application. For most of the application software packages, functional testing involves testing the Graphical User Interface (GUI). For example, for a payroll application, through the GUI, it needs to be tested whether the software is able to generate the payroll correctly for the employees. The results obtained through the software have to be compared with the existing manual system. If the results match, then we can confidently say that, the software is indeed meeting the functional requirement. Non-functional requirements are generally divided into three categories [Sommerville 1996]:

y Product requirements y Process requirements y External requirements Figure 3.2 shows the various requirements under these three categories. Non-functional requirements

Product Requirements

Reliability

Portability

Process Requirements

Usability

Efficiency

Performance

Delivery time

Space

Implementation requirements (hardware & software requirements)

External Requirements

Standards

Safety Response Time

Throughput

Privacy/ security

Inter Operability

Ethical Legal

Figure 3.2 Non-functional Requirements In the requirements engineering phase, the engineers will find out which of the nonfunctional requirements are applicable to this particular project/product on hand; and

Fundamentals of Testing

105 only those will be mentioned in the SRS document. For example, suppose the customer wants library automation software to be developed by your organization. The customer wants the software to run on Windows XP environment. However, your management may feel that you should develop the software in such a way that it works on both Windows XP and Linux environments—the management wants to make money by selling the product to other customer as well. In such a case, portability is one of the nonfunctional requirements. The customer does not care whether the software is portable or not; only you should test the software for portability.

If portability is a requirement, the various platforms on which the software has to work also will be specified (for example, Windows XP and Red Hat Linux operating systems running on Pentium 4 HT processor). The test engineers need to test the software on both these operating systems. The test engineers need to study the various non-functional requirements that need to be tested and write test cases for each of these requirements as well. Note that testing each non-functional requirement is a very involved task. The software has to be tested for both functional requirements and nonfunctional requirements. The quality requirements given in the quality triangle give the various non-functional requirements of software.

3.8 Testing Strategies There are two types of testing:

y Static testing y Dynamic testing

3.8.1 Static Testing In static testing, the software is tested without executing it. This is generally done by manual code reviews and code inspections. The programmer, along with other group members, goes through the code and checks for any syntax errors and logical errors. The code is checked to ensure that it implements all the requirements given in the SRS document. It is also checked whether the code has enough comments so that other engineers can easily understand the code, for easy maintenance. These code reviews can be done informally or formally. This is called static testing or static analysis.

Chapter 3

106 Static testing can be done not only on the code, but also on other work products, such as SRS document, design document, test plan, etc.

Static testing is generally done manually. Tools are also available for static analysis. For example, whether the code written in C programming language is portable or not can be easily tested using a tool ‘lint’ available on the UNIX systems. In static testing, the code is checked without executing it. Static testing is done through code reviews and code inspections, which are carried out informally as well as formally.

3.8.2 Dynamic Testing In dynamic testing, the code is executed and tested. The code is compiled, executed with test inputs, and if the actual results match with the expected results, then the software can be considered as OK. This sounds very easy, but what inputs need to be given is a major issue. Consider a program to find the square root of a given number. You first write the code and then execute it. You need to give some numbers as input and check whether you are getting the correct result or not. The inputs can be: 12, 34, 16, 32, 0, -23, -46,…………

For positive numbers, it should give the correct value (±4 for 16. It should give 0 if the input is 0 and if the input is a negative number, it should display a message: “Please give only positive numbers”. Now, can we say that the software is perfectly OK? Suppose, you give a number 45678990. What will be the output? If your program takes integers as input, and if the system allocates only 2 bytes to an integer, then the above value is not acceptable as it results in an overflow and you will get a run time error. So, there are lots of issues that need to be resolved in dynamic testing: you need to check the software for valid inputs and invalid inputs, and also ensure that the software gives appropriate error messages. Testing is of two types: static testing and dynamic testing. In static testing, the code is tested without executing it. In dynamic testing, the code is executed and the actual results are compared with the expected results.

3.8.3 Why Testing is Difficult? In dynamic testing, the code is executed and the test inputs are given then the actual results are compared with expected results. How many types of test inputs have to be given is the major issue. Ideally, we need to give all possible test inputs.

Fundamentals of Testing

107 Exhaustive testing Consider a simple example: to test a program which takes three numbers (for three sides of a triangle) as input and checks whether the sides form a triangle or not. How do we test the program? We have to give different combinations of inputs (three numbers) and check whether the program is working correctly. We need to give correct inputs (numbers which form a triangle) and wrong inputs (numbers which do not form a triangle) and ensure that the output is as expected. To give the correct numbers again, the numbers can be integers, floating point numbers or a combination of both. We also need to give characters and check the program (if you give single characters, it gives interesting output!). We need to give small integers (say, 10,12,3) and also large integers (568888, 9999999, etc). We need to give some inputs which have one or two 0’s; and all zeros, to check how the program is functioning. So, to carry out an exhaustive testing of such a simple problem, the input data to be given will be enormous. It takes nearly two hours to give all possible combinations and ensure that this program is ok! Testing is difficult because:

y We need to test the software for both right and wrong inputs and check the functionality as well as the performance parameters.

y We need to give the inputs in such a way that each and every line of the code is tested.

y We need to give inputs randomly and check that the software never fails. y We need to test the software as though a normal user is using it and check whether the necessary error messages, help, etc. are provided.

y We need to test the software in different hardware and software environments depending on the requirement.

y We need to test the functional and non-functional requirements. As the complexity of the software increases, the time taken to do such exhaustive testing becomes enormous and exhaustive testing becomes humanly impossible. To summarize, testing is difficult! To carry out the testing in a limited time with limited effort and resources, and to bring out all the possible hidden defects is a challenging and intelligent task. Let us consider another example of testing a GUI of a standard calculator (available on your Windows systems (Start→Programs→Accessories→Calculator). The GUI is shown in Figure 3.3.

Chapter 3

108

Figure 3.3 GUI of Standard Calculator In this GUI, the meaning of various symbols is as follows:

y y y y y y y y y y y y y y y y

+: To perform addition -: To perform subtraction *: To perform multiplication /: To perform division .: Decimal point sqrt: To find square root of a number %: To find percent 1/x: To find inverse of a number MC: To clear the memory MR: To recall from memory MS: To save in the memory M+: To add to the memory C: To clear the current calculation CE: To clear the displayed number Backspace: To remove left most digit +/-: To give sign to a number (positive or negative)

Your job is now to test whether the calculator is working correctly or not. You have to test the addition (+) by adding integers, floating point numbers—small numbers, large numbers—and check whether the results are OK or not. You need to repeat this process or all other arithmetic operations. Then you need to check whether correct percentages are being obtained. You need to test whether the inverse number functionality is correct. You need to check all memory operations. You need to do all this

Fundamentals of Testing

109 for positive numbers, negative numbers, 0, very small numbers, very large numbers and so on. For one small GUI like this, if you have to give all possible combinations of inputs and test, then you need infinite time and energy to do the testing. Here’s yet another example to show that exhaustive testing is practically impossible. You need to test a program that checks whether the input filename is correct or not. As per the specifications, the filename can have 1 to 8 letters (A to Z), only in upper case. Digits and any symbol or special character is not allowed. Blank in between letters is also not allowed. The various inputs you need to give are filenames with one letter, two, three, four, five, six, seven and eight letters. Each letter can be one of the 26 alphabets in upper case. These are all the valid filenames. So, the number of valid test inputs is: 26 + 262 + 263 + 264 + 265 + 266 + 267 + 268.

In addition, you need to test the program by giving at least some filenames with nine letters, some filenames containing spaces, special symbols and digits. For these invalid inputs, the software must display an error message indicating that the filename is not valid. So, you can conclude that this type of exhaustive testing is not of any practical importance. You should never even attempt to do it. What is the alternative? You can give only a subset of all the valid input filenames (with 1 to 8 characters) such as: a, bc, def, ghij, klmno, qprstu, vwxyzab, cdefghij

You can also give a few filenames from the set of all invalid filenames (with nine characters, space in between, containing special symbols) such as: pqrstuvwx, gh c, gh$k

If you test the program with only a limited number of test inputs, you still get confidence that the software is OK. In exhaustive testing, all possible inputs are given to check whether the program works correctly. As exhaustive testing is time consuming and practically impossible for large programs, the software has to be tested only by giving selected inputs which are likely to uncover the bugs in the software, i.e. to give only a subset of the possible inputs and test the software. So, one fundamental problem with testing is that software can be never completely tested. In spite of our best efforts, we may still leave behind a few defects when we deliver the software to the customer. How sad! But it is a reality

Now the test engineer’s job is reduced to a single task: select the possible test inputs that will bring out maximum number of defects in the software. Such test inputs are called test cases.

Chapter 3

110 The test engineer’s main challenge can be defined as: to design the test cases such that these test cases bring out the maximum number of hidden defects in the software. An efficient test case design will reduce the testing time and effort considerably.

3.8.4 Test Case For the designer: Bring up Test Input and Test Result beside the arrows When you want to test a particular specification of your software, you have to give certain inputs to the software when it is being executed. For these test inputs, you expect some output (the expected result). But actually, the software will produce some output (the actual test result). Note that the expected result and the actual test result will be same if the software is working as per the design. Otherwise, they will differ. Just before giving the test inputs, the software will be in a particular state. This initial state is described by entry criteria. After the testing, the software will go to another state. This state is described by exit criteria. A ‘test case’ defines all these aspects. As shown in Figure 3.4, a test case consists of:

y y y y

Entry criteria (the pre-conditions before the test inputs are given) The test inputs The exit criteria (the post-conditions after the test inputs are executed) The expected results Entry Criteria

Exit Criteria

Executable Code Test Input

Test Result

Figure 3.4 Test Case Test case and test input are generally used interchangeably. You need to remember that the test case consists of test input as well as the entry/exit criteria and expected results

Suppose you need to test a program that takes a filename as input, opens the file, writes today’s date into the file and closes the file. You can define a test case as follows:

y Test input: filename “abc.dat” y Entry criteria: The file abc.dat should not exist in the current directory.

Fundamentals of Testing

111 y Expected output: The file abc.dat should be in the current directory, with the contents in today’s date.

y Exit criteria: The program should terminate without errors. The file abc.dat should be available in the current directory.

You need to define a number of such test cases for testing all the specifications of your software. As shown in Figure 3.5, a number of test cases are selected, the program is executed and the results are compared with the estimated results.

Select Test Cases

Execute Test Cases

Analyze Test Results

Figure 3.5 Testing Process using Test Cases Consider the example of sorting a set of given strings. The string can be simply letters, or words or sentences. All these possible strings have to be given as inputs and the software has to be tested. It has to be ensured that the software does not malfunction (or the system does not crash) when a large string is given as input. So, you need to select some test inputs from the set of all possible test inputs and design the test cases. Designing the test cases is the most challenging assignment of test engineers. Test cases have to be designed based on two criteria: reliability and validity. A set of test cases is considered to be reliable if it detects all errors. A set of test cases is considered as valid, if at least one test case reveals the errors. Test engineers’ challenge lies in uncovering as many defects as possible with minimum number of test cases. Test case design/selection is very crucial for making the project a success. In every software project, time and budget are always limited. In a short time, carrying out a thorough testing is possible only when you select the test cases that reveal the hidden defects. During the system testing, the testing has to be done repeatedly. For example, whenever the code is modified, testing has to be done. So, the test cases need to be repeatedly executed. If you have a large number of test cases, then again, a lot of time is required for testing. Always, the test engineers need to strive to minimize the test cases but at the same time, to detect the defects fast. A test case consists of the test input, the entry criteria, the exit criteria and the expected output. After executing the test case, the expected result is compared with the actual result which indicates whether the software is functioning as desired or not.

Chapter 3

112 Pesticide Paradox In 1990, Boris Beizer, used the phrase pesticide paradox to describe an interesting phenomenon in testing. When you use the same set of test cases to test software, you cannot uncover the defects. This is similar to using the same pesticide again and again— the insects get immune to it! Similarly, the software gets immune to your test cases. Hence, it is not a good strategy to repeatedly use the same set of test cases. You need to keep on changing the test cases.

Defect Clustering In a software package consisting of a number of modules, it is likely that defects cluster in a module or function. So, when you encounter defects in a module, keep testing that module more rigorously by choosing a large number of test cases that can test that module. This can happen due to various reasons—the developers who worked on that module would not have got the necessary expertise, the module would have been more complex, etc. Whatever be the reason, if you find a good number of defects in a module, thorough testing of the module is a must. More test cases need to be selected to test the functionality of the module. While selecting the test cases, pesticide paradox and defect clustering need to be kept in mind. Just as insects get immune to a pesticide when it is repeatedly used, the software gets immune to a test case when the same test case is used repeatedly. To avoid this pesticide paradox, the test cases need to be changed. Defect clustering indicates defects cluster in one or few modules, and those modules need thorough testing.

3.8.5 Test Oracle Oracle is a repository of knowledge. In earlier days, people used to go to a place, known as an oracle, where the God is depicted in the form of a statue. The people go beside the statue and convey their problems and the God (the priest who used to hide beneath the statue) used to answer the person and give a solution. Test oracles are the people (or machines) that check the correctness of the program for a given test cases as shown in Figure 3.6. Human test oracles are used extensively, if the program does not work. A common way is to give the program and the input test data to a friend and ask him to derive the output from the program. Your friend acts like a computer, goes through the code step by step as per the input data, and arrives at the output. If this output is same as the expected, then the program can be said to be correct, otherwise there is a defect in the software.

Fundamentals of Testing

113 Software under test Comparator

Test cases

Test oracle

Results of testing

Human oracle

Figure 3.6 Test Oracle Many software projects are now being undertaken which are for replacing an old system with a new one. The old system may be a manual system or a software-based system. When the new software is developed with the same functionality, you can compare its output/results with those of the old system. If the results match, then you can say that the new software is working OK. In this case, the old system is a test oracle. The automated testing tools act as Test oracles. Whatever the human oracle does, it is automated through a software program.

3.8.6 Test Software Test engineers need to appreciate that the environment in which the software is developed may be different from the environment in which the software will be actually used by the customer. In such cases, you need to simulate the actual working environment in your laboratory. This can be done by writing test software called simulator. Here are some examples:

y Suppose you need to test the software that has to work on a satellite-based

computer network. Since you will not have access to a satellite-based network, you can simulate the environment on a LAN or by interconnecting the computers using serial communication links. You need to write some additional software to simulate the long delays in a satellite network (the computer to computer delay over a satellite link may be up to two seconds). If the software works in the simulated environment, you get some confidence on the software. Finally, you need to conduct a field trial by installing the software on the satellite-based computer network and testing it.

y Suppose you have to test an embedded software that has to be ported on an ARM processor based hardware module. If the module is not available to you, you need

Chapter 3

114 to test the software on a PC using the cross platform tools and emulators supplied by the processor/embedded software vendor.

y Suppose you have to integrate your software package with another software

package being developed by another organization. When your software is ready, you can write special software that gives the necessary inputs from the other software package. In other words, you are simulating an environment in which a subsystem can be tested. This is called test harness. Writing test harnesses is a part of test engineer’s job.

y Testing unit/components is also very difficult for developers. Additional code needs to be written to test the units/component. This test software is called stubs and drivers.

3.8.7 Manual vs. Automated Testing For many years, software testing was done manually. In manual testing, the test engineers’ job will be as follows:

y From the specifications, design the tests y Design the test cases y For each test case, ensure that the necessary test setup is available and the entry criteria are met

y y y y

Give the test inputs Check whether the test results are matching with the expected results If there is deviation, it is a defect. Enter the defect details in a database Repeat the entire process for all the test cases

This manual testing has a lot of problems. Every time the testing is repeated, manually entering the data becomes very boring and it is likely that mistakes are made in data entry. Comparing the actual results with expected results may also turn out to be boring for many types of testing. Entering all the details of the defect, each time, is again a very tedious task. Lot of repetitive, unproductive work can be easily automated using software testing tools. Nowadays, a lot of commercial and open source tools are available that can be used for carrying out functional testing and performance testing. Tools are also available for managing the entire testing process. Specialized testing tools are used for testing embedded systems, networks, etc. We will discuss the details of testing tools in Chapter 8.

3.8.8 Testing Software of Different Technologies The principles of testing are the same, but applying these principles to test different technologies comes with experience. In your testing career, it is likely that you may need

Fundamentals of Testing

115 to test software for different technologies and hence it is necessary to gain sufficient knowledge in those technologies and also learn the intricacies of testing a software of these technologies.

y Client/Server software: Client/Server software is the most widely used software

nowadays. Using stand-alone applications is rather limited. In Client/Server systems, the resources available on the server are accessed by multiple clients simultaneously. Test engineers need to keep in mind that all clients may not use the same hardware and software and in spite of the differences, the software should work. Even when multiple clients access the same resource (say, a database) simultaneously, the response should be very fast. If the system becomes slow, it is not acceptable. The communication link between the client and the server also affects the performance. Security is another important feature to be tested in client/server systems.

y Web-based applications: Internet and Intranet based applications are now gaining

popularity. Whether it is testing a simple web site or a complicated web service or an e-commerce application, the following are the issues: the web application has to work with different browsers—most applications are tested only for one browser and for other browsers, the display just looks bad. Each and every link on every page needs to be tested. Each page download should not take more than 10 seconds. The audio/video need to be handled with additional care. Security is the most important issue for e-commerce applications.

y Object-oriented software: Testing software written in an object-oriented

programming language poses special challenges. Re-use is the most attractive feature of object-oriented programming languages. So, it is likely that some portions of the present code have been developed elsewhere, and used in other projects. Adapting the code written for some other project has to be done very carefully. Similarly, the code that is being given to other groups for adaptation has to be tested thoroughly and given.

y Mobile applications: Mobile communications is the latest technology fad. Mobile

phones and smart phones are being used not just for voice communication but also for email and Internet access. Mobile commerce, mobile learning, etc. are now the applications being offered on mobile devices. Developing mobile applications is different from developing desktop applications—small display size, low bandwidth of mobile network, high delays and different mobile communication protocols are the issues to be tackled. Testing the mobile applications also involves testing in a simulated environment using mobile phone emulators and then deploying the application over the network for field trial and testing the application. Micro-browsers available on the mobile phones, small display size with different resolutions, easy navigation of the menu items and so on, are the focus of testing mobile applications.

Chapter 3

116 Every new technology poses special challenges not only in development, but in testing as well. The test engineers need to gain insight into the new upcoming technologies regularly.

3.9 Metrics in Testing Phase During every phase of software development, metrics data need to be collected. The metrics help in monitoring the progress of the project and also in improving the process. Some important metrics in testing phase are given below:

y Estimated time for the testing phase and the actual time taken for the testing phase. During the planning stage, the time required for the testing phase is estimated based on past experience or using historical data. While executing the project, the actual time taken for the testing is measured. If the two values are close to each other, then it can be concluded that the estimation process is good enough. Otherwise, the estimation process needs to be improved.

y Estimated budget for the testing phase and the actual money spent on the testing phase. These values also help in improving the estimation process.

y Estimated effort for the testing phase and the actual effort. The effort is measured in person months.

y Defects density. The number of defects found per 1000 lines of code is called defect density. Defect density gives an indication of the quality of the software.

y Number of defects found per unit time (say, a week). During the testing phase, the

testing team has to keep track of the number of defects found per week (or per day or per fortnight, depending on the duration of the testing phase). Tracking this data helps in finding out when testing can be stopped.

y Cost of quality. The three components of cost of quality are failure cost, appraisal cost and prevention cost, as discussed earlier.

Collection of metrics data during the testing phase helps in producing high quality product and also in improving the testing process.

3.10 When is Testing Complete? Surprisingly, it is very difficult to say when the testing phase is complete and when the product can be released. As you keep testing the software, it may happen that you keep getting a defect--and this process may go on forever. On the other hand, you may not detect any defect, but what is the guarantee that there is no hidden defect? And, that defect may show its effect on the day the software is delivered to the customer. See the

Fundamentals of Testing

117 irony, after all the effort of developing the software, you are not sure whether it is ready for shipment! This proves our point once more: testing is difficult. Instead of relying on gut feeling, we need to arrive at a scientific method to declare that the testing is complete and the product is ready for shipment.

3.10.1 Criteria for Completion of Testing It is very difficult to say when testing is complete. Three criteria used in practice for completion of testing [Pressman 1994] are:

y When you run out of time. y When you run out of money. y Based on statistical criteria. In practice, the first two criteria are followed. During the planning stage, certain time and money (or effort) is allocated for the testing process. The test team keeps testing the software, and when they run out of time (or money), the product is delivered (to avoid the penalty for late delivery). Most managers do not realize that this is a very dangerous practice because if the software fails at customer’s site, the reputation of the organization is at stake. A more practical approach for declaring that the testing is complete is to use the third criterion. After the coding is completed, and testing begins, initially many defects are detected. Slowly, the number of defects found in a given time (say, in a day or week) keeps on reducing. The graph showing the number of defects found every week would be as shown in Figure 3.7. If the number of defects found per week remains less than a pre-defined threshold consecutively for three weeks, then the software can be considered a mature product and released to the client. Number of defects found per week

Threshold

Week

Figure 3.7 Completion of testing based on defects found per week

Chapter 3

118 When testing is complete, it has to be to decided based on statistical criteria. The number of defects found per unit time (say, week) has to be recorded and if this value is below a pre-defined threshold consecutively for say, three weeks, the software can be considered to be mature enough to be delivered to the client.

Defect Seeding Another method used deciding when to finish testing uses defect seeding. In defect seeding, intentionally some defects are introduced in the code and the testing team is asked to detect the defects. Suppose, 100 defects are intentionally introduced in the code and the testing team is able to detect 90 defects. We can conclude that 10% of the defects are undetected, out of the intentionally introduced defects; so it is likely that 10% of the actual defects are also not detected by the testing team. Based on this conclusion, we can estimate the residual defects (or latent defects) in the software. In other words, we can conclude that: Total defects in software/total defects found = total defects intentionally seeded/seeded defects found (Total defects found + Latent defects)/total defects found = total defects intentionally seeded/seeded defects found (1 + latent defects/total defects found) = total defects intentionally seeded/seeded defects found Latent defects/total defects found = (total defects intentionally seeded/seeded defects found) –1 Latent defects = Total defects found * (total defects intentionally seeded/seeded defects found –1)

Suppose, during the testing you have found 300 defects in the software. When you intentionally introduce 100 defects, you are able to find only 90 defects. So, you can estimate the latent defects to be Latent defects = 300 * (100/90 –1) = 300 * (10/90) = 34.

However, note that calculating the latent defects in this fashion may not give the correct picture about the risk associated with the latent (or hidden) defects. One major hidden defect may crash the software, which is not acceptable at all.

3.11 Risk-based Testing In management, one of the important lessons we learn is profit is the reward for risk-taking. If you do not want to take any risk, you need not be in business at all. Even every software project will have risks associated with it. But then what is risk? Risk can be defined as probability of occurrence of a problem in future. Occurrence of that problem may cause a damage. Suppose you developed an e-commerce portal for on-

Fundamentals of Testing

119 line purchase of books. If the portal is made up without enough testing on the security aspects, then the probability of occurrence of security violations is very high and hence the damage will also be very high. So, while testing the software, we need to reduce the risk. This outlook towards testing is called ‘risk-based testing’. Intuitively, we all do risk-based testing. If you realize that a particular module has more bugs, you check it more—this effectively tries to reduce the risk of leaving some bugs in that module.

Mathematically, we can define risk as follows: Risk = Probability of occurrence of problem (P) * Damage that can be caused by the problem (D).

Note that it is very difficult to quantify risk. How much money you will lose if your ecommerce portal has security bugs? Difficult to say, but sometimes based on gut feeling, you may indicate a figure—US$ five million. A common way of measuring the risk is through grades—very high, high, low, and very low, etc.

3.11.1 Types of Risks Broadly we can divide risks into two types:

y Product risks y Project risks The product risks are due to leaving bugs in the software and delivering the software to the customer. These bugs can be related to the operational, revision, and transition characteristics given in the quality triangle. Some product risks are:

y Will the product work on different flavors of Windows? (Windows 95/NT/98/2000/XP)

y Will this software interface with the legacy database? The project risks are due to the problems in executing a project. Failure to complete the project within the time frame (time overrun) or within the budget (budget overrun) are project risks. If the project is not completed within the time frame, there are two types of damage: 1. You need to spend additional money due to the extended time on manpower to complete the project. 2. The customer may impose a late-delivery-clause and cut some money from your total project cost. Such damages are very costly and project risks need to be minimized to a great extent. Time and budget are the two constraints which limit the rigor, with which you can do testing. Hence, keeping in view the project risks, we need to devise a testing strategy so that the product risks are also minimized. And, that is the greatest challenge to the test engineers and test managers. Some of the project risks are:

Chapter 3

120 y y y y y y

Shortage of skills of team members Internal politics Lack of management support for testing/QA Third party supply failure Persons leaving in the middle of the project Accepting the project without studying the requirements well

Risk-based testing involves the following steps:

y y y y y

Identify the various risks for the project—product risks and project risks Analyze the constraints Based on the constraints, prioritize the tests to be carried out Identify the types of testing to be done Identify whether use of testing tools will be of help in reducing the risks In risk-based testing, project risks and product risks are identified and based on the estimated impact (or damage) of these risks, the testing strategy is worked out by working out what types of tests to be done, priority of the tests and whether the testing tools need to be used, etc. Project risks are also referred to as process risks. Product risks are specific to the software under development. Process or project risks are due to the constraints imposed by the process or management or external factors.

3.12 The Myths and Realities of Testing There are some myths prevalent amongst the prospective test engineers. Here are some of them:

y Testing is easy: Many young engineers fresh out of school think that testing is a

very easy job and they prefer a more challenging job ‘development’. It needs to be appreciated that testing is equally challenging job and a creative bent of mind is required to be a test engineer.

y Test engineers have no career growth: Some test engineers, when they begin their

career, look at their jobs as a stopgap arrangement. They do not look at it as starting a new career. Test engineers can have a career as lucrative as a development engineer. If you keep upgrading your skills continuously, career growth is no issue. Test engineers become better developers as they can bring out quality software. Test engineers can take up software quality assurance as their career option, which involves development of processes for the entire organization.

Fundamentals of Testing

121 y If I learn a testing tool, I will get a job: Learning how to use testing tools is an

important skill for prospective test engineers. However, that alone is not enough. You need to have a good understanding of the software quality assurance, learn the testing process thoroughly, gain expertise in at least one application domain and gain experience by working hands-on on a few testing projects.

y Testing does not involve coding: With the availability of testing tools, carrying out

testing repeatedly using the same test cases has now become easy, as no coding is required. However, to do advance testing, a number of test scripts have to be written. Writing test scripts is similar to writing code in any programming language. And, you also need to think: how long you keep coding in your life, you got to grow up!

y Testing is a phase that comes at the end of development life cycle: If testing is

taken up very late (when you are left with 15 days to deliver the software to the customer), then it is like taking a cancer patient to a doctor when the disease is out of control. Testing must be started at very early stages. If a module is ready, test it without waiting for others to complete their modules and then integrate them.

y Testing is completed: When you start testing the software, you find many defects; and slowly with rigorous testing, the number of defects found keeps reducing. Even if you do not find a single defect even after say, one week of testing, it does not mean that the software has no defects. Perhaps you have not designed the correct test cases!

y Developers know that their software will have bugs: Unfortunately, no software

developer thinks that his/her code has any defect. If you tell him/her that the software is not working properly (and also show it), even then, they will not agree. They will tell you 100 reasons why it did not work at that time: may be it is because of the keyboard or because of the problem with the operating system or something else! It is because the developers get emotionally attached to their work! It is sad, but it is a fact.

y Test engineers are the most-disliked people in the organization: In most

organizations, test engineers and quality assurance people are indeed disliked. This is because developers do not like their mistakes being pointed out by someone else. If you point out a problem, you become the bad guy. So, development group and the testing/QA group always fight with each other. But the test/QA engineers have to strive to be liked. This can happen if the test engineers communicate effectively. It is an art to point out a problem without hurting the person. If you cultivate that habit, you will be a great test engineer (and it also helps you in becoming a great husband/wife).

Chapter 3

122

Summary y The errors made by human beings while developing software results in defects or bugs in the software. The defects cause the software to fail.

y The objective of testing and debugging is not just to find the defect and rectify it, but also to find the root cause of the defect.

y To test the effectiveness of the testing process, sometimes bugs are intentionally introduced. This is called defect seeding or debugging.

y Software testing process can be defined as: the process that reduces the risk associated with residual defects.

y Validation is to check whether the software meets the customer expectations. Verification is to check whether the software conforms to specifications.

y Validation answers the question: Are we building the right product? Verification answers the question: Are we building the product right?

y Quality does not come free; a price has to be paid for it. It is called cost of quality. It consists of three components: failure cost, appraisal cost and prevention cost.

y It is advisable to have a separate testing team to test the software. The psychology of testing comes into play when the development team tests the software.

y In buddy testing, two persons pair together and do the testing, preferably a developer and a tester.

y The developer needs to be involved in unit/component testing and integration testing. System testing can be done independently by the test team.

y The V model depicts the various types of testing to be done at various stages of software development.

y The software product has to be tested for both functional requirements and non-

functional requirements. The non-functional requirements are divided into product requirements, process requirements and external requirements.

y There are two types of testing—static testing and dynamic testing. In static testing, code is not executed—the code is checked through reviews and inspections. In dynamic testing, the code is executed, test inputs are given and the expected result is compared with the actual result.

y It is practically impossible to carry out exhaustive testing—to test the software for all possible inputs and test conditions.

y In practice, a subset of all possible test cases is chosen and testing is carried out. A test case consists of the test inputs, expected results, the entry criteria and the exit criteria.

Fundamentals of Testing

123 y If you keep testing the software with the same set of test case, the software becomes immune to the test cases. This is known as pesticide paradox. So, it is advisable to design new test cases periodically.

y It is likely that defects cluster around one or a few modules. Testing needs to be done keeping in view the defect clustering phenomenon.

y While doing the testing, test software may needs to be written such as simulators, drivers, test harnesses, and stubs.

y It is difficult to tell when testing is complete. Generally it is done based on statistical criteria—if the number of defects found per week is below a threshold for say, three consecutive weeks, then the software can be considered mature.

y In risk-based testing, the possible product risks and project/process risks are

identified and the damage that can be caused by these risks is estimated. The types of tests to be done and the test cases to be executed are prioritized based on the risk analysis.

Review Questions 1. Intentionally introducing bugs in the software is called: a) Debugging

b) Defect clustering

c) Defect seeding

d) Error detection.

2. In Dynamic testing, code is not executed. a) True

b) False

3. To avoid pesticide paradox: a) Test cases need to be repeatedly used b) Test cases need to be changed regularly c) Test cases need to be used only once d) Test cases not to be changed 4. Exhaustive testing means: a) Testing the software for all possible combinations of inputs b) Testing the software using selected test cases c) Testing the software for parameters d) Testing the software by undergoing operational testing 5. While selecting test cases both valid inputs and invalid inputs have to be chosen a) True

b) False

Chapter 3

124 6. Persons leaving in the middle of a project is a: a) Product risk

b) Project risk

c) External risk

d) Internal risk

7. Whether the software will work on different flavors of Windows is a: a) Product risk

b) Project risk

c) Defect risk

d) External risk

8. Testing and debugging are one and the same. a) True

b) False

9. Usability is a: a) Product requirement

b) Process requirement

c) External requirement

d) Internal requirement

10. Safety is a: a) Product requirement

b) Process requirement .

c) External requirement

d) Internal requirement

11. When is testing complete? a) When you run out of time b) When you run out of money c) When the number of defects is below a threshold for a specific period. d) When you run out of both money and time. 12. Human action that caused a software failure is called: a) Error

b) Defect

c) Bug

d) Failure

13. Validation is same as quality assurance. a) True

b) False

14. Verification is same as quality assurance. a) True

b) False

15. Cost of checking whether the software has defects is called: a) Failure cost

b) Appraisal cost

c) Prevention cost

d) Probability cost

16. Cost of fixing defects is called: a) Failure cost

b) Appraisal cost

c) Prevention cost

d) Probability cost

17. Cost of changing the process to avoid defects is called: a) Failure cost

b) Appraisal cost

c) Prevention cost

d) Probability cost

Fundamentals of Testing

125 18. Which of the following work products has to undergo static testing: a) SRS document

b) Design document

c) Source code

d) All of the above

19. In which development life cycle model, testing is done most rigorously: a) Prototyping model

b) Waterfall model

c) Spiral model

d) Synchronize-and-stabilize model

20. If static testing is done on the source code, dynamic testing is not necessary. a) True

b) False

21. GUI testing can be done exhaustively. a) True.

b) False

22. The existing manual system can act as a test oracle when the manual system is being automated. a) True

b) False

23. If 10% of the seeded defects are discovered by the testing team, and a total of 500 defects were found in the software, the number of residual defects is: a) 50

b) 45

c) Cannot be estimated

d) 30

24. Risk is inversely proportional to the probability of occurrence of the problem. a) True

b) False

25. Risk is proportional to the damage caused by the problem. a) True

b) False

26. Metrics help in: a) Improving process quality

b) Improving product quality

c) Both (a) and (b)

d) Neither (a) nor (b)

Index

388

Chapter

Testing Levels and Types Chapter Objectives

4

y Levels of testing y Testing approaches y Types of testing

To bring out a quality software product, the developers need to test the software at different levels during the coding/testing phase. For example, without testing the individual modules, if the modules are integrated and system testing is done, it is extremely difficult to find out where the defect is. After the software is developed, different types of testing have to be done, keeping in view the functional and nonfunctional requirements of the software. In this chapter, we will discuss about the different levels and types of testing.

4.1 Testing Levels During the design stage, the software is divided into components or units and each component/unit is developed. This divide-and-conquer strategy is to make life simple. Similarly, while testing, it is no good if you combine all the units into modules and all the modules into the system in one shot and then start testing the entire system. A practical approach is to divide the testing process into different levels. To start with, each component/unit has to be tested separately; the modules have to be built from the units, which are then tested. The modules are then combined together and the system is built and tested As shown in Figure 4.1, each level of testing is for testing a specific entity. Unit testing is done to test the source code. Integration testing is done to test the design. System testing is done to test the SRS. And, finally, Acceptance testing is done to test the client/user requirements.

Chapter 4

128 Testing in Actual Environment

Acceptance Testing

System Testing

Integration Testing

Unit Testing

Business / Operational Needs

User Requirements

SRS

Design

Code

Figure 4.1 Levels of testing Unit testing is done to test the source code. Integration testing is done to test the design. System testing is done to test the system against SRS. Acceptance testing is done to test the software against client/user requirements. To check whether the software meets the operational/business needs of the client, the software is installed in the actual working environment and tested.

4.1.1 Unit/Component Testing A unit/component is the smallest entity in the software. Every unit has to be tested separately to check whether it is as per the specifications. As it is not possible to test a unit individually, additional piece of code may be written to test the units. Each function can be considered as a unit. Unit/component testing is done only by the development engineers. This testing is done to check the functionality as well as to ensure that the code is well written. Suppose you have implemented a stack. You have to write a main function, a push function to push an element into the stack and a pop function to pop an element from the stack. You can consider each of the three functions as a unit. Initially, you may just write the code for main function. How do you test it without push and pop functions? Instead

Testing Levels and Types

129 of the statement that calls the push function, you can just write a statement that prints call push here. Similarly, you can write a print statement that prints call pop here in the place of the statement that calls pop function. If the main program works, then you can develop the push function, replace the above print statement with the call to push function, and then test the main program along with the push function. Then you can develop the pop function and test the program for entire stack implementation. Testing of a collection of units/components that have a logical relationship is called stream testing. Suppose you have developed the code for a few data structures (a stack, a queue and a tree), you may like to test all these units one after the other—this is string testing. It may not be possible to test a component/unit in isolation. You may need to write additional code (stubs, drivers, simulators) to do the testing.

4.1.2 Module Testing A module is an independent entity in the software. The tested units are integrated into a module and each module is tested separately for the specifications. For instance, if the software is a database management system, the modules can be database (back-end) and the user interface (front-end). The database can be tested separately using the SQL commands to check correct data retrieval. However, validation of the front-end (forms) requires a database. So, after testing the back-end module, the front-end can be tested, after integration. After completion of testing a module, a module-level test report has to be prepared. If the modules are independent, these modules can be tested in parallel. Testing multiple modules/sub-systems in parallel is called parallel testing. In a modular design, if the modules are designed to be independent to a great extent, it helps in parallel testing. Parallel testing greatly reduces the testing time.

4.1.3 Integration Testing After the modules are tested, the modules can be integrated together. It is a difficult task to integrate all the modules together and start testing, as it would be very difficult to do the debugging. The integration has to be done systematically by incremental building and testing in steps. During integration testing, a very systematic approach has to be followed. Foremost, we need to identify which modules can be integrated first. The interfaces between the modules should be well defined to achieve this. For very large projects involving large number of modules, incremental building is the most challenging task. Once all the modules are integrated together, the system is ready for testing.

Chapter 4

130 Consider an e-learning portal. Some important modules in this portal are given in Figure 4.2. Note that each module is for a specific functionality. Broadly, the portal consists of three modules: admin module, tutor module and student module. E-Learning Portal

Admin Module

User Management

Student Module

Tutor Management

Tutor Module

Student Management

Email

Chat

On-line Content Content Management

On-line Tests

On-line Tests Allocation

Program Monitering

Figure 4.2 Various modules in an e-learning portal The student module in turn consists of a number of sub-modules, such as chat for online chatting with other students, email, on-line content viewing which consists of powerpoint presentation with explanation and voice-over, and on-line test which is for taking an on-line multiple choice test. The tutor module contains sub-modules for content management (adding new lecture material), on-line tests allocation for various students and progress monitoring. The admin module consists of sub-modules, such as user management (user names and password), student management (enrollment, certification, etc.) and tutor management (enrollment, allocation of courses, etc.). During the construction/implementation stage, all the sub-modules (or components) are developed. Now how do you integrate them? You can start integrating the sub-modules of the student module, build the complete student module and then test it. Similarly, you can integrate and test the tutor module and the admin module. After each of the modules is tested to a great extent, integrate all the three modules and build the complete portal. Though bottom-up method and top-down method are advocated for integration, the best approach is to study which modules can be independently tested to a great extent; integrate the various components of that module and test the module.

Testing Levels and Types

131 Without following any systematic approach, if you integrate all the modules together in one shot and then start testing, it is called big bang testing. The problem with big bang testing is that debugging is very difficult—if the software does not work, how do you find out in which module the defect is

4.1.4 System Testing Once all the modules are integrated, the system is ready for a full-fledged testing. The various approaches used for system testing are discussed in the next section. The various types of testing the system has to be subjected to, are given subsequently. To start with, the system has to be tested for its functionality. This is done by going through the SRS document and for each requirement, carry out the test cases design, execute the test cases, and compare the expected results with the actual results. This functional testing, obviously, differs from application to application. Hence, applicationdomain knowledge is a must for test engineers. If you are an expert in testing banking software, you will not know the intricacies of testing a telecommunication software package. In addition to functional testing, the system has to be tested for its non-functional requirements, such as:

y y y y y y y y y

Performance requirements Accessibility to physically handicapped persons Portability Security Reliability Safety Conformance to national/international standards Interoperability with other hardware/software Conformance to any legal/statutory requirements

We will discuss how to test non-functional requirements in the section on types of testing.

4.1.5 Acceptance Testing After the system testing is completed in your premises, the software has to be approved by the customer. The customer will carry out a rigorous testing—called Acceptance testing—and if there are no errors/defects found, then the customer will pay your money. Acceptance testing can be done either at the developer’s or client premises. So, we need to be very careful about this testing phase and do a rigorous planning to achieve success in the first round of acceptance testing. If we encounter problems during this

Chapter 4

132 testing, the customer will be unhappy, and he will carry out the testing more rigorously next time. The various stages of testing are shown in Figure 4.3. For each stage of testing, a test plan has to be prepared and after the testing is done, a test report has to be prepared. Figure 4.4 describes pictorially the complete testing process as discussed in this section along with various test plans to be written at each level. Unit Testing

Unit Testing

Module Testing Module Testing

Sub -system Testing Sub -system Testing System Testing Acceptance Testing

Figure 4.3 Stages of testing

Testing Levels and Types

133 Unit Test Plan

Unit Test

Combine Unit into Modules

Module Test Plan

Integration Plan

Module Test

Interconnect Modules

Acceptance Test

System Test Plan

System Test

Acceptance Test Plan

SRS

Client Requirements

Figure 4.4 Levels of testing and corresponding test plans Depending on the client requirements and contractual obligations, the acceptance testing may be of different types:

y User acceptance testing: The user acceptance testing is carried out at the developer’s/user’s premises based on an acceptance test plan.

y Operational acceptance testing: The software is installed in the actual working environment and the performance observed over a period of time to check how the software meets the users’ operational needs.

y Contractual fulfillment testing: If there are any special contractual obligations, such as environmental testing for the hardware, testing the system for air-borne applications (such as in aircraft), then testing is done to fulfill these obligations.

y Regulatory acceptance testing (or conformance testing): The software may be

required to conform to international standards or meet some statutory government requirements. In such a case, the necessary testing is done. This testing may be done at third party laboratories or government laboratories.

Chapter 4

134 During the project proposal stage, the type of acceptance testing to be done has to be finalized as it will have a major impact on the time frame and budget. A common mistake is that the customers only ask for user acceptance testing in the lab; and when the software is installed in the actual working environment, it creates problem. It is advisable to insist on operational acceptance testing—it is good for both developer and customer

4.2 Testing Approaches In a large software project, it is impractical to integrate all the modules in one shot and start testing the software as a whole. The system has to be built in stages and the product has to be built incrementally carrying out testing on each incremented software. Such a systematic approach helps in easier debugging. The testing approaches discussed in this section help in achieving the objective.

4.2.1 Static Testing vs. Dynamic Testing After the code is written, there are two ways of testing this code—static testing and dynamic testing. In static testing, the code is checked without executing it; hence it is also known as desk checking. This can be done either formally in a meeting or informally by two or more development/QA engineers. In dynamic testing, the code is executed and the actual results are compared with the expected results. It is advisable to carry out both the types of testing because each testing reveals specific bugs. For instance, if the development team forgot to implement a particular user requirement, this can be revealed only in static testing as the code is checked for completeness. But dynamic testing does not reveal such an error. In static testing, the code is checked without executing it through reviews/inspections. In dynamic testing, code is executed and checked. Both types of testing are required to be done.

4.2.2 Positive Testing vs. Negative Testing The software has to do, what it is intended to do and nothing more or nothing less. We need to test the software to check whether it is doing what it is intended to do. But that is not enough. We also need to check whether it is doing anything, which it is not intended to do. This is differentiated by positive testing and negative testing. Positive testing is testing the software to ensure that it does, what it is intended to do. Negative testing is testing the software to ensure that it does not do anything; it is not intended to do.

Testing Levels and Types

135

4.2.3 Top-down Testing vs. Bottom-up Testing In top-down approach, testing is done from the top of hierarchy. Dummy routines called stubs are introduced that simulate a module. In bottom-up approach, testing is done from the bottom of hierarchy. Dummy routines called drivers are introduced that invoke a module. Consider testing the software that implements a stack. The program contains a main function and functions to push and pop. In top-down approach, first the main function is tested. For this, in the place where a push function is called, a print statement can be introduced which prints call the push function. Similarly at the point where a pop function is called, a print statement can be introduced which prints pop function is called. In bottom-up approach, a driver function is written, which just calls the push function first and then testing is done to ensure that the push works well. Similarly, the pop function is tested. And subsequently, all the functions are combined together and tested. At system level, carrying out top-down testing is difficult, it is better to do the testing using bottom-up approach—test the sub-modules, the sub-systems and the total system. In top-down testing, testing is done from the top of hierarchy. Dummy routines called stubs need to be written in this approach. In bottom-up approach, testing is done from the bottom of hierarchy and dummy routines called drivers need to be written. Generally, the top-down approach is followed for design and bottom-up approach for testing.

4.2.4 Functional Testing vs. Structural Testing In functional testing, what the software does is tested. Here, the functionality of the module is tested and structure is not considered. Test cases based on specifications and internals of modules are also not considered. This type of testing is also known as black box testing. A structure of a program differs at different levels. At the component level, the structure is the code. At integration level, the structure is the tree diagram that shows the interconnecting modules. At system level, the structure is the menu structure, or web page structure or a business process depending on the application. Structural testing is also called clear-box testing, glass box testing and white box testing.

Chapter 4

136 Structural testing at component level is used to test the implementation of the program. Here, the source code is looked into for testing the software. Structural testing involves: a) statement coverage b) condition coverage c) path coverage

Statement Coverage This ensures that each and every statement is executed. It is advisable to ensure that every line in the code is executed at least once. Checking it manually is difficult, but tools are available for this. Profilers used to carry out this statement coverage testing. Profilers indicate the number of times each line in the code is executed. When you execute a test case, the profiler indicates which lines of code have been executed and which lines have not been executed. After studying the output of the profiler, you can create new test cases that execute the earlier unexecuted lines.

Condition Coverage In branch coverage, each and every branch option (or every condition) is taken, and inputs are given in such a way that, each branch is executed at least once. For example: if (a > 6 && b < 5) then ; else ;

Here, it has to be certainly ensured that both the ‘then’ statements and ‘else’ statements are executed at least once. The test cases are chosen in such a way that the condition becomes true and also false. In addition, it is necessary to ensure that each individual condition in the conditional statement is made true and false, i.e. a > 6 is true and false; b < 5 is true and false. Condition coverage in percentage = (no. of conditions executed/total conditions) * 100

Path Coverage In path coverage, it is ensured that each path is executed at least once. You can convert a given program into a flow chart and you will know pictorially how many paths are there. You need to design test cases to ensure that each path is tested at least once. As an example, consider the flow chart shown in Figure 4.5(a) and the corresponding flow path given in Figure 4.5(b). Here, there are two paths to be tested. Hence, test cases need to be formulated to ensure that both the paths are tested.

Testing Levels and Types

137 Find d

Y

Is d > 0?

N

Calculate

Calculate

Real Roots

IMG Roots

Stop (a)

(b)

Figure 4.5 (a) Flow Chart; (b) Corresponding Flow Diagram Note that path coverage deals with the logical paths in the program. For small programs, the path coverage can be easily ensured by designing the necessary test cases, however in practical systems with millions of lines of code, 100% path coverage is not that easy. Path coverage in percentage = (no. of paths executed/total paths) * 100

The flow diagram shown in Figure 4.5 (b) can be used to find the cyclomatic complexity of the program. If E is the number of edges, N is the number of nodes; cyclomatic complexity is E – N + 2. In Figure 4.5 (b), E = 4, N = 4 and hence cyclomatic complexity is 2. If cyclomatic complexity is more than 20, the program is considered to be complex and the testing becomes very challenging.

Function Coverage In function coverage, it is ensured that each and every function is called at least once. This ensures that all the functions in the program are tested. Developing test cases for function coverage is rather easy as compared to developing test cases for path coverage and condition coverage.

Chapter 4

138 In functional testing, software is tested for its functionality without considering the implementation details. In structural testing, the implementation details are taken into consideration. Structural testing at code level involves statement coverage, branch coverage, and function coverage. 100% decision coverage (IF statements) will guarantee 100% statement coverage, but not vice versa. 100% path coverage will not guarantee 100% condition coverage.

Structural Testing at Integration Level Consider the various modules of an e-learning portal given in Figure 4.2. In this software, when we carry out the testing at module level, we can consider the various sub-modules in that module and then work out the testing strategy. In the student module, the chat, email and on-line content sub-modules can be tested independently. However, to test the on-line tests sub-module, we need to also consider the on-line tests allocation submodule of the tutor module. So, at the testing level, we can consider the structure of the modules for finalizing the testing strategy.

Structural Testing at System Level When you consider the structure at the system level, you look at the major modules in the total system. For instance, in the e-learning portal, there are three modules: admin module, tutor module and student module. You can to start with test the student module, then the tutor module and finally the admin module. In the case of a web page, the home page of the web site gives the overall structure of the web page—the main links in the web site. When you click on each link, you get the sub-links. In top-down structural testing, you can begin at the home page, and go down to the leaves. In Figure 4.6, the home page of a web site that provides weather information is shown. You can easily make out the structure of this web site: there are 3 links for weather resources: weather basics, weather phenomenon and weather forecasting. You can search the site by city name; and also select a city and obtain the weather information. Once the overall structure is known, you can click on each link and get the internal details. After the overall structure is obtained, you can start the testing from the home page (top-down approach) or the bottom-most links (bottom-up approach).

Testing Levels and Types

139

Figure 4.6 Home page of a Web site

Black Box Testing vs. White Box Testing at Code Level Both black box testing and white box testing have to be done for every software package. White box testing presents some advantages in detecting the defects as well as reducing the testing time/effort as illustrated in the following examples. Suppose you have written a function fn(x) as follows: fn(x): if (x > 0) return abc(x) otherwise return pqr(x)

Chapter 4

140 Here, the return value is abc(x) if x > 0 and the return value is pqr(x) if x 0, abc(x) function will be called and only when x 0 and pqr(x) for x 5 && b 0) { root = sqrt(d); } if (d > 0 { root1 = -2 * a / d ; root2 = 2 * a/d ; } else { printf(“No root\n”); } do { statements; } while ( x > 0);

(binary operator)

Chapter 5

178 In GNU software, you can have the utility called “gindent” (GNU indent) for indentation of the source code.

y Whenever changes are made to a source file, modify the header indicating the date of modification and why the modification has been done.

/* Revision history */ /* Date: 17 July 2006 max_temperature initialization value changed to 55 as per client requirement */

y It is advisable to use the source code configuration software (sccs or rcs, or any other equivalent software) if there are a large number of source code files to be maintained.

y During the course of development, the code written by one engineer should be

given to another engineer who has to check whether the coding guidelines are being followed or not. Peer reviews are an effective way of doing informal code review. The code written by one engineer should be reviewed by another engineer to check readability, understandability and maintainability. Subsequently a formal code review can take place.

5.3.4 Code Optimization For many applications, there will be stringent performance requirements. In some embedded systems, a particular task has to be completed in a specified time, otherwise, it will lead to a catastrophe. In some systems, there will be strict memory constraints and the programmers have to take into consideration the two important performance constraints--speed of execution and memory requirements. Writing the code in assembly language increases speed of execution and also results in optimal code. However, maintaining the code written in assembly language is difficult. Hence, the code is generally written in a high level language such as C or C++ and then some portions of the code, which take lot of execution time, are written in assembly. Even while, writing the code in a high-level language, code optimization can be done to make the execution faster. Some important code optimization guidelines are given below. While carrying out the code review, you need to keep these guidelines in mind:

y Eliminate dead code: Dead code is that portion of the code that will not be

executed at all. Novice programmers include such code, declaration and initialization of variable that will not be used at all, use of functions that will not be called at all etc. Profilers help in elimination of dead code.

y Remove unnecessary debugging code: Lot of code is written during the

development time for debugging purposes. Unnecessary debugging code needs to be removed. However, note that some debugging code may need to be left out in the final software that helps in debugging the system in the field.

Static Testing Techniques

179 y Avoid recursion: Recursive functions require a large stack resulting in higher

memory requirement. For example, instead of using recursion for calculating a factorial of a number, a better approach is to use a lookup table that will pick up the factorial.

y Avoid floating-point operations: If necessary, floating-point operations can be converted into fixed-point operations.

y Use unsigned integers: Use unsigned integer variables wherever possible. y Avoid large library routines: To the extent possible, large library routines need to be avoided.

y Remove loop-invariant code: The code that has no effect on the loop counter is called the loop-invariant code. Consider the example: for (i=0; i < 10; ++i) { x= 1.0/b; y = x + i; }

In this code, the statement x=1.0/b; is loop-invariant because the value of x is not dependent on the value of i. Hence, this statement can appear above the beginning of the for loop. Otherwise, the value of x is calculated every time resulting in unnecessary execution of a statement. Elimination of dead code, avoiding recursion, avoiding floating-point operations to the possible extent, avoiding large library routines and removal of loopinvariant statements are some of the important guidelines for code optimization. Though compilers provide an option for setting optimization level, formal code review is must because the compiler can not take care of all the optimization requirements given above.

5.3.5 Java Coding Guidelines This section gives the guidelines to be followed by all software engineers while doing programming in Java. These guidelines have been framed to ensure that the code can be understood easily by persons other than the original author of the software, for easier maintenance. As the objective of coding is not to reduce coding time or effort but to reduce testing and maintenance effort and time, all engineers are advised to follow these guidelines.

Chapter 5

180 Backup Policy and Schedules during Development Phase During the development phase, the backup activity of the unit/module level code is the responsibility of the individuals. At the end of each day, the software has to be backed up in the specified folder on the server. This ensures that at most only one day’s effort is lost in case of a system failure.

Project Folder Each software developer will have a folder with the name of the project in his/her directory on his/her machine in the root directory. In this folder, a file with the name ‘readme’ should be present which gives a description of all the sub-folders and file names in the folder corresponding to the project. The version number of the software should be reflected as a part of the filename. For example, quest1.0 should be the file name for the quest project version 1.0. All the names of the directories and files should be meaningful indicating the module name or the unit name. In case of abbreviated names, the ‘readme’ file should clearly explain the contents of the directory or file. A logical hierarchy should be maintained, with one folder for each module, with files corresponding to each unit/function within the module. The ‘readme’ file should be in the format: Filename

description

File name extensions will be compiler/programming language/OS.

status (tested OK, testing in progress)

as

per

the

standards

required

by

the

Once a program module tested and found OK, the ‘readme’ file should be updated with the new status. If older versions of the module need to be kept, they should have a different extension (eg. v0.9) and listed in the readme file.

Java Coding Conventions The header should be in javadoc format. Use documentation comments immediately before declarations of interfaces, classes, and member functions. Documentation comments are processed by javadoc. /** Subscriber: A subscriber is a person who uses The IVR system @author KVKK Prasad */

The important javadoc tags are given below:

y @author name: This is used for classes and interfaces to indicate the author(s) of the code. One tag has to be used per author.

Static Testing Techniques

181 y @deprecated: This is used for classes and member functions to indicate that the API for the class has been depreciated and should not be used any longer.

y @exception name description: This is used for member functions to describe the

exceptions that a member function throws. One tag per exception should be used and full class name for the exception has to be given.

y @param name description: This is used for member functions to describe a

parameter passed to a member function, including its type/class and its usage. One tag per parameter has to be used.

y @return description: This is used for member functions to describe the return value, if any, of a member function. The type/class and the potential uses of the return value should be indicated.

y @since: This is used for classes and member functions to indicate how long the item has existed since JDK 1.1.

y @see ClassName: This is used for classes, interfaces, member functions and fields.

It generates a hypertext link in the documentation to the specified class. You should use a fully qualified class name.

y @see Classname#memberfunctionName: This is used for classes, interfaces,

member functions and fields. It generates a hypertext link in the documentation to the specified member function. Use a fully qualified class name.

y @version text: This is used for classes and interfaces to indicate the version information for the code.

Naming Methods Methods should be named by using a full english description, using mixed case with the first letter of every non-initial word capitalized. Examples:

y y y y

openConnection() printAddressLabel() save() delete()

This convention, though increases the typing effort of the programmer, enhances readability and hence understandability of the code as the purpose of the method is immediately known.

Naming Getters Getters are the methods that return the value of a field. Prefix the word ‘get’ to the name of the field. If it is a boolean field (returns true or false), use the prefix ‘is’ to the name of the field instead of ‘get.’

Chapter 5

182 Examples:

y y y y

GetLastName() getAccountNumber() isPersistent() isAtEnd()

Naming Setters Setters are the methods that modify the values of a field. You should prefix the word ‘set’ to the name of the field, regardless of the field type. Examples:

y y y y y

setLastName(String aName) setAccountNumber(int anAccountNumber) setReasonableValues(Vector newValues) setPersistent(boolean isPersistent) setAtEnd(boolean isAtEnd)

Naming Constructors Constructors are the methods that perform the necessary initialization when an object is first created. Constructors are always given the same name as their class. For example, a constructor for the class Patient would be Patient(). Examples:

y Customer() y SavingsAccount()

Naming Fields (Fields/Properties) Fields that are collections, such as arrays or vectors, should be given plural names to indicate that they represent multiple values. Examples:

y y y y

lastName zipCode unitPrice orderItems

Naming Components/Widgets Components/Widgets have to be named appropriately (instead of button1, button2, raju, etc.) so that, it will be easy to identify their purpose and type.

Static Testing Techniques

183 Examples:

y y y y

okButton customerList fileMenu newFileMenuItem

Naming Constants In Java, constants are typically implemented as static final fields of classes. The convention is to use full words, all in uppercase, with underscores between the words. They ensure that there are no magic numbers. When constants are initialized to specific values, indicate why you are initializing to that value in comments. Examples:

y MINIMUM_BALANCE y DEFAULT_START_DATE

Naming Local Variables A local variable is an object or data item that is defined within the scope of a block, often a method. The scope of a local variable is the block in which it is defined. In general, local variables are named following the same conventions used for fields i.e., by using full descriptors with the first letter of any non-initial word in uppercase.

Naming Streams When there is a single input and/or output stream being opened, used, and then closed within a method, the common convention is to use in and out for the names of these streams, respectively. For a stream used for both input and output, use the name inOut. An alternative to this naming convention is to use the names inputStream, outputStream, and ioStream instead of in, out, and inOut respectively.

Naming Loop Counters Since the scope of the loop counters is limited, use counters such as i, j, k. Counters such as counter1, counter2 are appropriate for member functions.

Naming Exception Objects Use the letter e for generic exceptions.

Naming Parameters (Arguments) Convention to be followed for parameters is same as for local variables. Use full description of the value or object being passed, prefix the name with the letter ‘p’ to indicate that it is an argument.

Chapter 5

184 Examples:

y pCustomer y pInventoryItem

Naming Classes, interfaces and packages y Naming Classes: Use a full English descriptor starting with the first letter capitalized using mixed case for the rest of the name. Examples: Customer  Employee  Order  OrderItem  FileStream  String 

y Naming Interfaces: Use mixed case with the first letter of each word capitalized.

The preferred convention for the name of an interface is to use a descriptive adjective, such as Runnable or Cloneable, although descriptive nouns, such as DataInput or DataOutput are also common. Examples:  Runnable  Contactable  Prompter

y Naming Packages: Use full description of the package appropriate to the main function. Put all classes that can be grouped together into their own packages. Use sub-packages for smaller components of a big project.

Examples:  package Forms;  package Widgets;

What to Document The following guidelines are to be followed to document the various items:

y Arguments/Parameters: The type of the parameter, any restrictions or preconditions, the significance of the parameters.

y Fields/Property fields: Description, documentation of all applicable invariants, concurrency issues and visibility decisions.

y Classes: Purpose of the class, development and maintenance history of the class, applicable invariants, concurrency strategy and known bugs.

Static Testing Techniques

185 y Compilation units: Each class/interface defined in the class, including a brief

description, file name and other identifying information, copyright information and name of the author. The ratio of commented lines to non-commented lines is a coding metric that indicates the amount of documentation embedded in the source code. A ratio of 1:1 is considered good i.e., if there are 1000 lines of non-commented lines, there should be 1000 lines of comments. Many development engineers think this is crazy, but the SQA people know better and let us face the fact: the SQA team will have the final word.

5.4 Static Analysis using Tools Though manual review is a good mechanism for static analysis, a number of tools are available, particularly for analysis of the source code. These tools will make our lives easy by pointing out a number of defects in the code such as:

y y y y

Whether the code is as per the coding standards/guidelines. Syntax mistakes. Declaring a variable and not using it. Unreachable code (also called dead code). Code which will never be executed is called unreachable code.

y Security vulnerabilities of the code (such as buffer overflow). y Portability problems (if the code contains machine dependent or operating system dependent features, then the code is not portable).

y Coding metrics such as:  



Total number of lines. Number of commented lines to the number of uncommented lines (in some organizations the ratio should be 1: 1). Code complexity (number of function points, number of function calls etc.).

A number of tools are also available that assist in designing, particularly when you use an Object Oriented Programming language like C++ and Java. In such case, OOA and OOD tools such as UML tools will come in handy. IBM Rational’s tools belong to this category. Static analysis tools are used extensively by the developers during the development phases such as design, implementation, and integration testing. To give you an idea of the tools, we will discuss two simple utilities available in the Unix/Linux environment in this section.

Chapter 5

186

5.4.1 Tool for Readability Improvement/ Indenting When you write lengthy programs, indenting the code increases the readability. The shell command indent provides the automatic indenting feature. To indent a C program, say hello.c, the shell command is as follows: $indent hello.c

The following options are available. Option

Description

-bad

Blank lines after indentation

-bap

Blank line after each function/procedure body

-bbb

Blank line before boxed comments

-sob

Remove unnecessary blank lines in the source code

-bl

Braces after if line

-bli

Brace indent

-bls

Braces after structure declaration line

-br

Braces on if line

-nbbo

Break after Boolean operator

-bbo

Break before Boolean operator

-clin

Case indentation

-cn

Comment indentation

-nsc

Do not star comments

In some Unix systems, the equivalent command is C beautifier (cb). The command format is: $cb hello.c The SQA team has to make sure that the source code written by the development engineers is maintainable. One important feature of maintainable code is readability. If the source code has proper indentation, readability will be more.

5.4.2 Portability Testing Tool To test the portability of C programs is important when the software has to be ported from one platform to another, for example from Unix system running on a Pentium platform to a Solaris system running on SUN SPARC platform. The “lint” utility available on the Unix/Solaris system can be used to check the portability of the code.

Static Testing Techniques

187 In addition to portability problems, lint also gives messages regarding “bad programming style”. The various messages given by lint are:

y y y y y y y

Unused variables and functions. Assigning a ‘long’ variable to an ‘int’. Variables which are used first and set later. Unreachable break statements. Function that returns a value which is never used. Type casting problems. Non-portable character use [for example, if c is a character variable, the code if((c = getchar() < 0) is not portable].

y Unusual constructs [for example *p++; does nothing]. y Control statements that never succeed [if x is an unsigned integer, if(x < 0) is never a success].

The command format of lint is as follows: $ lint hello.c

The output will display the potential portability problems. We will show the lint output for very simple files. You can pass your C programs through the lint utility and study the output messages to see what kind of good (or bad?) code you write. Here is a small C program with an include directive and the main function in which a variable is declared and printf function is called. #include void main(void) { int i; printf("Hello\n"); }

Let us pass this program through lint utility by giving the command: # lint -n test.c

Output: test.c (as included in test.c) ================= (4) warning: variable unused in function: i in main name declared but never used or defined _iob stdio.h(170) value type declared inconsistently _lastbuf llib-lucb:stdio.h(172) struct __FILE * :: stdio.h(172) struct __FILE *

Chapter 5

188 fopen stdio.h(274) struct fdopen stdio.h(274) struct freopen stdio.h(274) struct popen stdio.h(274) struct fread stdio.h(280) int () fwrite stdio.h(280) int () fopen64 stdio.h(326) struct freopen64 stdio.h(326) struct

llib-lucb:stdio.h(207) struct __FILE *() :: __FILE *() llib-lucb:stdio.h(257) struct __FILE *() :: __FILE *() llib-lucb:stdio.h(208) struct __FILE *() :: __FILE *() llib-lucb:stdio.h(258) struct __FILE *() :: __FILE *() llib-lucb:stdio.h(239) unsigned long () :: llib-lucb:stdio.h(240) unsigned long () :: llib-lucb:stdio.h(323) struct __FILE *() :: __FILE *() llib-lucb:stdio.h(324) struct __FILE *() :: __FILE *()

function returns value which is always ignored printf

You can easily make out the problems associated with the above program by studying the lint output. Now let us slightly change the program, and run the lint utility again: void main(void) { int i; printf("Hello\n"); }

Let us pass this program through lint utility by giving the command: # lint -n test.c

Output: test.c (as included in test.c) =================

(3) warning: variable unused in function: i in main (4) warning: implicitly declared to return int: printf function returns value which is always ignored printf Now, let us revise the program again slightly: void main(void) { printf("Hello\n"); } Let us pass this program through lint utility by giving the command: # lint -n test.c Output:

test.c (as included in test.c)

Static Testing Techniques

189 ================= (3) warning: implicitly declared to return int: printf function returns value which is always ignored printf If the SRS document specifies that portability is a requirement, then the test engineers can use the lint utility to check the portability related issues. If the programming language used is C, it is always a good practice to use the lint utility because it analyzes the code and shows, if the programming style is bad.

5.4.3 Symbolic Execution Symbolic execution is also known as symbolic evaluation. In this method, the program is executed with symbolic data. For instance, consider the following program segment: int x; int i1, i2 ; x = i1 + i2;

For i1 and i2 variables, instead of giving the actual values, symbolic values such as p and q are given and the results are computed. In this case, the value of x should be p + q. This type of testing is very useful in some specialized applications that involve lot of mathematics. Tools are available that carry out symbolic execution as manual testing by using this approach is very difficult. Let us test a conditional statement as: if condition then statements

The condition is first assumed to be true and symbolic execution is done. Then the condition is assumed to be false and then symbolic execution is done. This type of testing is very useful for long logical expressions. Symbolic execution has limitations—overflow cannot be detected. It is also not very suitable for loops. Hence, only portions of the program can be checked and hence scope of this type of testing is very limited.

Summary y In static testing, software work products are reviewed and source code is analyzed without executing it. In contrast, in dynamic testing, the code is executed and checked.

y Static testing is done to find defects in the early stages of development. In dynamic testing, the test engineers find failures.

Chapter 5

190 y Manual review is the most effective way of static testing. Manual review can be

done on any work product such as SRS document, design document, test plan, test cases, test scripts, acceptance test procedure, user manual etc.

y In formal review process, a moderator leads the meeting. The other members of the review team are: author of the work product, experts in the application domain/business process/programming language and a secretary.

y Slight variations to the formal review meetings are informal reviews, walkthroughs and inspections.

y Source code must be subjected to formal code review to check for completeness, missing functionality, adherence to coding guidelines etc.

y Every organization has to develop a set of coding guidelines to be followed for each programming language.

y Checklists and guidelines must be used in the review meetings for effective reviews.

y If the software does not meet the desired response time or if the execution is very slow, code optimization needs to be done.

y Static analysis tools are used by the developers to model the system, to carry out

the integration testing, to check whether the code is as per the coding guidelines or not.

y The ‘lint’ utility can be used to test whether the C code is portable or not. This utility also gives the information about bad programming style such as unused variables, unused functions etc.

Review Questions 1. Checklists help in: a) Static testing b) Dynamic testing c) Both static testing and dynamic testing d) None of the above 2. Whether the code is as per the coding standards is checked through: a) Static testing

b) Dynamic testing

c) Parameter testing

d) Global testing

3. You can always rely on code optimization done by a compiler: a) True

b) False

4. A work product may be reviewed several times in static analysis: a) True

b) False

Static Testing Techniques

191 5. In walkthroughs, the team is led by: a) Author

b) Moderator

c) Project manager

d) Programmer

6. Inspections are done through: a) Peer reviews

b) A team of experts

c) A team of programmers

d) A team of reviewers

7. The ratio of commented lines to uncommented lines should be preferably: a) 1:1

b) 1:2

c) 2:1

d) 1:10

8. Indentation of code helps in: a) faster execution

b) better readability

c) less code size

d) None of the above

9. Dead code means: a) code that gives runtime errors b) code that gives compilation errors c) code that is not executed at all d) None of the above 10. Recursive function requires: a) large stack

b) higher processing power

c) small stack

d) No stack

11. In the following code snippets, find out the defect: void main() { int myarray[10]; … myarray[10]= 10; …. }

12. In the following code snippet, find out the defect: void main() { long int k; int x; ….. k = x; }

Chapter 5

192 13. In the following code snippet, find out the problem from optimization point of view: for( k = 0; k < 10; ++k) { p = 3.15 * x * 82.456; …… }

14. In the following code snippet, find out the problem: unsigned int x; …… if ( x < 10) { …….. }

15. In the following code snippet, find out the possible error: int m = 2, n = 3; float p; p = m/n; printf(“%f\n”, p);

16. In the following code snippet, find out the value of k: int m = 10, n = 5, p = 2; float k; k = m/n*p;

Now, replace the last line with m/(n*p) and find out the value of k. Why do the values of k differ in both the cases?

Chapter

Dynamic Testing and Test Case Design Techniques

6

Chapter Objectives y y y y y y

Dynamic testing methodology Test specifications and test design Techniques for test case design Test case documentation Test case execution and incident reporting Defect logs

The process of testing is similar to the process of development. You need to work out the test specifications and design, carry out the detailed test case design, document the test cases, execute the test cases, report the results of each test case execution and log the defects. In this chapter, we will study the details of this testing process and the various techniques used for designing the test cases. We will illustrate the test case design techniques with case studies.

6.1 Dynamic Testing In dynamic testing, the code is executed and checked for any defects. This testing has to be done very systematically and involves the following steps:

y y y y y y y y

Review work products Identify test objectives Identify test specifications and carry out test design Design the test cases Document test cases Execute the test cases Generate incident report Log the defects

Chapter 6

194 The process of testing is similar to the process of development—you need to work out the test specifications and test design, carry out the detailed test case design, execute the test cases and generate the defect logs.

6.2 Review Work Products When a software product is given to you for testing, before taking up the actual testing, you need to study the work products that have been written during the development phases. These work products include:

y y y y

SRS document Design document Source code Draft user manual (if available)

From these documents, you need to develop the test specifications. The entity based on which you develop a test specification is called “test basis”. Test basis can be a functional specification, a design strategy or a portion of the code found in the work products. To start with, you quickly go through the work products to get an overview of the software, but you need to keep referring to the work products as you keep looking for test basis for writing the detailed test specifications. The detailed test specifications are developed based on the ‘test basis’ which can be a functional specification, a design strategy or a portion of the code, which are available in the work products. Some tools give information such as consumption of processor power, consumption of memory, using and freeing of memory, memory leaks etc. This is called dynamic analysis of the program and tools that provide this information are called dynamic analysis tools. Dynamic analysis provides the inputs for designing the test specifications.

6.3 Identify Test Objectives After you get a good feel of the software, the next step is to study the details of the quality requirements of the product and identify what quality parameters need to be tested. Some quality parameters are:

y y y y

Usability Reliability Performance Code coverage

Dynamic Testing and Test Case Design Techniques

195 y y y y y

Portability Security Safety Conformance to a standard External interfaces (with hardware or software)

If you have to test software that is being developed in-house or through a subcontractor, you need to do the testing a number of times till the software is accepted by the customer. In case, you are testing an off-the-shelf software package (say, a payroll package or an ERP software), your job may be just to test whether the software meets your requirements or not. In either case, you need to define the objectives, but testing offthe-shelf software is not as complicated as testing the in-house developed software. Test objectives define the various quality parameters that need to be tested. If you have to test commercial off-the-shelf software, you have to only test whether the software meets your needs or not. In the case of in-house developed software, your testing objective is to help the developers in bringing out a quality product that meets user requirements.

6.4 Test Specifications and Test Design Test specifications describe what to test. Test specifications are obtained based on the knowledge about the software, documentation available for the software and the risks to be covered. While carrying out a detailed review of the work products, you can come out with a list of test specifications based on the answers to the following questions:

y In addition to the functional requirements, what non-functional tests have to be done?

y Is it necessary to do the structural testing? y What types of tests are required in addition to black box testing and white box testing?

y If conformance testing is required, what are the details of conformance

requirements? In which lab, the conformance testing has to be done? What are the special requirements of conformance testing?

y What type of acceptance testing is required? Is it user acceptance, operational

acceptance, conformance acceptance or any special contractual obligations need to be met?

y If operational acceptance testing is required, how long the field trial or beta testing has to be conducted? In which location(s) it has to be done?

Recall that in risk-based testing, depending on the risks identified for the project, you need to prioritize your testing strategy. So, while working out the test specifications, you

Chapter 6

196 need to identify those tests that have to be done on a top priority, and the tests that may need to be done only if time/budget permit. For example, portability of software would have been identified as a quality parameter. However, if there is a time constraint, portability can be assigned low priority. The software can be tested on one platform and subsequently portability testing can be taken up. Once the test specifications are identified, the next step is test design which involves:

y Based on prioritization of the test specifications, identification of test environment

(the hardware platform, operating system and other third-party software such as database engine, ERP software etc.).

y In case of embedded software, identification of the hardware platform on which the software has to be tested.

y Identification of test instruments such as oscilloscopes, spectrum analyzer, radio communication test set, power meter, frequency meter etc.

y y y y

Identification of the necessary tools to be used for the testing. Working out test case specifications and test data. Test script generation. Working out which of the test cases need to be used in regression testing keeping in view pesticide paradox.

y Working out the details of software stimulators/driver software/stub software to be written.

Test specifications and test case specifications need to be differentiated. Test specifications give the overall requirements of the testing of the software. Test case specifications give the specifications of individual test cases for testing a specific functionality/quality parameter.

6.5 Design Test Cases Test cases have to be designed for each test specification. Note that for one test specification, there may be one or more test cases. Test case design involves the following steps:

y y y y y

Identify the test inputs. Identify the test data. Work out the test procedure. Write any test scripts, if required. Is there any special driver/stub software required? If so, write it.

Dynamic Testing and Test Case Design Techniques

197 y Identify the entry criteria. y Identify the exit criteria. y Identify the expected result (including any error messages). There are a number of techniques used for designing the test cases. We will study them in this section. Generation of the test cases is the most important and creative task in testing. Note that the testing tools increase the productivity of the test engineers many fold but the ingenuity of the test engineer is reflected in his/her design of the test cases.

Test cases have to be designed based on two criteria: reliability and validity. A set of test cases is considered to be reliable if it detects all errors. A set of test cases is considered as valid if at least one test case reveals the errors. Keep the following guidelines while designing test cases:

y Test case should be simple. For an input, there should be only one expected output. If there are two or more expected outputs, the test case becomes complex.

y y y y y

Test case should reveal errors. There should be no redundancy. You always need to take valid inputs and also invalid inputs. Test cases should closely match with the actual usage of the software. In addition to functional requirements, for each quality parameter also you need to design test cases.

y For designing test cases, the checklists, we discussed in Chapter 5, are very useful. Test case design techniques are broadly classified into the following categories:

y Black box test case design techniques. y White box test case design techniques. y Experience based test case design techniques. We will discuss the details of these categories.

6.5.1 Black Box Test Case Design Techniques In black box testing, we are only interested in the functionality of the software. So, we do not bother about the implementation issues. Test case design is effectively based on how the end user uses the application to carry out his/her activities with the software. However, the test cases have to be done for both positive testing and negative testing.

Chapter 6

198 Specifications Based Testing In this technique, the test cases are designed based on the specifications. For each specification, the input values are chosen and the output is checked. Example: The specifications of a program to calculate the roots of a quadratic equation ax2 + bx + c = 0 are as follows: read the values of a, b, c. If the roots are real, then print them. If the roots are imaginary, print the message “Roots are imaginary”. The program need not calculate the roots if they are imaginary. So, you choose some test cases such that roots become real and also some test cases such that the roots become imaginary.

y Test case 1: Test inputs: a = 1, b = -5, c= 6 Expected result: Roots are real, root1 = 3, root2 = 2

y Test case 2: Test inputs: a = 1, b= -4, c = 4 Expected result: Roots are real, root1 = 2, root2 = 2

y Test case3: Test inputs: a = 1, b = 4, c = 8 Expected result: Display of the message “Roots are imaginary”

Equivalence Partitioning In this technique, all the possible valid inputs and valid outputs are identified. The test cases consist of both valid inputs and invalid inputs. A group of tests forms an equivalence class if:

y All these tests test the same thing. y If one of the tests in the group catches a bug, then all the tests in the group will catch the bug.

y If one of the tests in the group does not catch a bug, then all the tests in the group will not catch the bug.

It is not only the equivalent input values, you can take equivalent output events or values, equivalent operating environments etc. and choose test cases from each class. Example: Consider a program that finds the square root of an integer. While testing the program, if the input is 4, suppose the correct result is obtained. You can generalize and say that for all positive numbers, the program works well. Suppose the input is –4 and the program gives a run time error. Then you can say that for negative integers, the program is written incorrectly. In this case, you have two equivalence classes: set of positive integers and set of negative integers. You need to select test cases from both the equivalence classes.

Dynamic Testing and Test Case Design Techniques

199 Example: Consider a program that takes three numbers as input and checks whether the numbers form a triangle or not. The two equivalence classes are: the set of numbers that form a triangle and the set of numbers that do not form a triangle.

y Test case1: Inputs: 3, 3, 3 Expected result: The message “The numbers form a triangle”

y Test case 2: Inputs: 3, 4, 5 Expected result: The message “The numbers form a triangle”

y Test case 3: Inputs: 0, 0, 0 Expected result: The message: “The numbers do not form a triangle”

y Test case 4: Inputs: 3, 1, 8 Expected result: The message: “The numbers do not form a triangle” Note that for all the above test cases, the entry criterion is that the program should wait for user input and the exit criterion is that the program should terminate/display the prompt for giving inputs.

y Test case 5: Inputs: 78000, 32000, 81900000 Expected result: An error message: “The numbers are too big and cannot be handled by the program”. Exit criteria: Program should terminate because of overflow. When the above test cases are executed, we can easily find out how good the programmer is—whether the program works as expected for both valid and invalid inputs. In the case of test case 5—how the overflow is handled must be made clear. One possibility is to prompt the user to give only numbers within a range (“only positive numbers below 30000”). Example: Consider the GUI of a database application as shown in Fig. 6.1. In this form, there are 4 fields to be entered by the user. “Emp No.” takes only integers, “Emp Name” takes a string, “Emp Salary” is also an integer and “Designation” is a string.

Chapter 6

200

Figure 6.1 Form for data entry Strictly, in the SRS document, the specifications of each field in the form need to be mentioned—in the case of employee name, what is the maximum size allowed, whether any special characters are allowed or not etc. If the SRS document does not give the details, you need to test the field by assuming what is valid and what is invalid. Some test cases for employee name are given below. Note that only the employee name is mentioned here. You can give valid inputs for all other fields and then test the following test cases.

y Test case 1: Employee name: Stan F. Fleming Expected output: A message indicating that the database is updated with new record. Exit criteria: The database should be updated with the new record.

y Test case 2: Employee name: “Stan F. Fleming” Expected output: An error message should be displayed indicating that there is an error in the employee name field. Exit criteria: The earlier state of database should be maintained. Note that when you enter a text field within double quotes, it will not be a valid field in the database (due to the syntax of the SQL command). You need to take special care in programming to make double quotes accepted in the field. Example: You need to test a Login screen as shown in Figure 6.2.

Dynamic Testing and Test Case Design Techniques

201

Figure 6.2 Login screen Note that many users can log in to the system by using this screen. So, it is not enough if you test for one user, you need to test for multiple users. Each user will have a username and a password. You need to give multiple usernames and corresponding passwords for testing this screen. Carrying out a test with multiple data values is called data-driven testing. Executing a test case with multiple data is called data-driven testing. The test data can be stored in a database and for each record in the database, the test case can be executed.

You need to create a database (say, using MS Access database or using Excel spreadsheet) and enter the data (username – password) in multiple records. You need to write a test script that takes each record, substitutes the values of username and password and checks whether the login is successful or not. You need to apply equivalence class partitioning here also and test for both valid and invalid inputs. Table 6.1 gives the test data for the test cases to check for both valid inputs and invalid inputs. Username

Password

Expected output

john

mary124

Login successful

JOHN

Mary124

Login unsuccessful

john

Mary124

Login unsuccessful

john

Login unsuccessful

fleming

Susanne

Login successful

SYSADMIN

mary124

Login unsuccessful

SYSADMIN

M1r2r4y

Login unsuccessful

Prasad

KVKK

Login successful

KVKK

Prasad

Login unsuccessful

Table 6.1 Test data for test cases of Login screen In equivalence partitioning, the test inputs are divided into valid inputs and invalid inputs. Test inputs are chosen from both valid inputs and invalid inputs. This approach is used in testing of validating data input, GUI input as well as in data-driven testing.

Chapter 6

202 Boundary Value Analysis In boundary value analysis, you need to identify the boundary values for the inputs at which the software is likely to fail. Generally, the programmer is likely to make mistake at boundary values and hence there is a high probability of detecting errors at boundary values. Example: Suppose you need to test the form, as shown in Fig. 6.1, for entering employee data in a database. You need to test the salary field. Suppose the salary field is declared as an integer that takes the values from 1,000 to 65,535 (a 2-byte positive integer). In this case, 65,535 is a boundary value. You need to give a valid boundary value (65,535) as well as an invalid boundary value (65,536) and then check whether the data is being accepted or whether an error message is being displayed. This is for testing the upper limit. Similarly, we need to test the lower limit of the salary. Here, the boundary is defined by 1000. So, we need to give 999, 1000 and 1001 and check whether the operation is correct or not. In case of 999 and 1000, the database should be updated, and in case of 1001, an error message should be displayed. Example: The boundary value concept can also be applied to string constants. Suppose the employee name should be at least 10 characters long and a maximum of 30 characters are allowed. You need to give test inputs 9/10 characters long (valid boundary value) and 11 characters long (invalid boundary value). Designing test cases using boundary value analysis is very important because the programmers are likely to make mistakes at the upper and lower boundaries of the values of the data.

Decision Tables Decision tables are used when there are many conditional executions. The test cases need to be selected to ensure that all the conditions are tested at least once. Example: Consider the income tax laws. The income tax to be paid depends on whether you are a male or female, whether you are a senior citizen (above 65 years) or not etc. Imagine a country in which the tax laws are as per the following table (in no ountry the laws will be that simple). Salary

Individual (male)

Individual (Female)

Senior Citizen

= or >. Perhaps the programmer made a mistake in typing >= instead of >. If you are a test manager and if you happen to work with engineers fresh from college, check every condition. The complexity involved in ensuring that each and every line of code is tested has to be appreciated. Windows XP has 40 million lines of code; for some versions of Linux, the code is nearly 200 million lines.

Structural Testing at Module Level When you integrate units into modules or classes into a functional module, then you can test the module taking into consideration the module structure. Example: Consider an object-oriented software for banking. As shown in Fig. 6.5, there is an abstract class called ‘account’ from which accounts such as savings account, current

Chapter 6

208 account and fixed deposit are derived. You cannot form an object of the type ‘account’ because it is an abstract class; but you can form objects of the type savings bank account, current account and fixed deposit. Account

Current account

Savings bank

Fixed deposit

Figure 6.5 Class hierarchy To test this module, you can form accounts of various types and check the methods/interfaces.

Structural Testing at System Level When you interconnect modules and develop the complete system, you will know how the modules are interconnected and hence using that information testing can be done. For example, Fig. 6.6 shows the structure of a web site. There are four links on the home page: About us, Products, Services, and Contact us. For each link, what are the next level links are also indicated. You need to test whether the same structure is indeed there in the web site. You also need to test all the links and ensure that there are no missing links. Home Page

About Us

Management

Services

Products

Workforce

Training

Telecom

Mobile Phones

Figure 6.6 Structure of a Web site

Consulting

Software

PBX

Payroll

Contact Us

CRM

Dynamic Testing and Test Case Design Techniques

209

6.5.3 Experience-based Test Case Design Techniques Experience-based test case design is more of an art than science. As you gain experience in testing, you become smarter in knowing the weaknesses of programmers and the requirements analysts. So, you design test cases keeping in view these weaknesses and generally you succeed.

Error Guessing Error guessing comes through experience. As you gain expertise and experience in testing, you can easily guess what types of mistakes, developers make in implementing a functionality and in writing code. Based on this ability to guess likely errors, you design test cases. Error guessing is simple test, based on your gut feeling. Highly experienced test engineers can detect many defects in a rather short time by using this approach.

Exploratory Testing Generally, while testing, we use the pre-defined test cases. This is called ‘scripted testing’. Testing carried out without using the pre-defined test cases is called exploratory testing. In exploratory testing, the test cases are designed and executed in real-time as against by using pre-defined test cases. Note that this is different from ad-hoc testing — ad-hoc testing is done rather unsystematically. Some process-oriented people do not like exploratory testing to be done. But it is a very useful technique. When you are testing the functionality of the software, you may suddenly feel the need to try out a new test case—and invariably the new test case will detect a defect. If the exploratory testing yielded a defect, then that test case can be documented. Based on the principles of test case design discussed above, we will work out the various test cases for (a) an Interactive Voice Response System and (b) a fingerprint recognition system in the following sections.

6.5.4 Case Study #1: Test Cases for an IVR System To test an Interactive Voice Response (IVR) system, we will design the various test cases keeping in view all the above principles of test case design and the various quality parameters that need to be tested. An IVR system consists of (i) hardware (a PC add-on card) to which two telephone lines can be connected, and (ii) associated application software. Assume that two users can access the IVR system simultaneously through their fixed/mobile telephones. The user can dial an IVR number to obtain the arrival/departure time of a train, by keying in the train number. The various types of tests to be done along with the test cases are listed in Table 6.3.

Chapter 6

210 Test types

Description

Functional testing

Test the system for valid train numbers. Test the system for invalid train numbers.

State transition testing

Dial the IVR number, get IVR response and disconnect. Dial the IVR number and disconnect. Dial the IVR number, key in train number and disconnect. [After the tests, ensure that the IVR system gets back to its normal state—ready to accept calls from two users.]

Structural testing

Draw the complete menu structure of IVR and test all possible paths.

Performance testing

Test the system when 2 users access simultaneously from their fixed telephones. Test the system when 2 users access simultaneously from their mobile phones. Test the system when 2 users access simultaneously, one from fixed telephone and the other from mobile phone.

Stress testing

Test the system when 3 users access simultaneously, two users should get IVR responses, third user should get an ‘engage’ tone.

Gorilla testing

Dial the IVR number and key in digits randomly—does the system withstand the random inputs or does it hang?

Experience based testing

When you use an IVR system repeatedly, you get used to the messages. So, you may like to key in the train number without waiting for the long message to be completed. The IVR system should be capable of accepting the digits even while a message is being played, and it should be capable of giving the arrival/departure information. IVR systems do not work with old telephone instruments (that support pulse dialing). Test the IVR system by dialing from an old instrument.

Reliability testing

Keep the IVR system on throughout the night and test the next day by using functional testing test cases. Switch off the IVR system (as though there is a power cut). Switch it on and test the system by using functional testing test cases.

Interface testing

The train arrival/departure information is stored in a database. Suppose it was earlier tested with MS Access database. Replace it with Oracle database and repeat the about test case execution.

Portability testing

Check the portability of the application software by using a portability testing tool. Check whether the driver for the PC add-on card for the other operating system is available (as the driver software is not portable).

Dynamic Testing and Test Case Design Techniques

211 Test types

Description

Security testing

A user can dial an IVR system, listen to the welcome message and then he doesn’t key in the train number nor does he disconnect. You need to check whether the IVR software will automatically detect this type of ‘misuse’. If two users do such a thing intentionally, the IVR system is not available for anybody else. This is called denial-of-service attack. The software has to be tested to check whether it is vulnerable to this type of attack.

Conformance testing

As the IVR system is connected to the telephone network, it should meet the requirements specified by the telecommunication authorities. However, only the hardware specifications will be tested—such as voltage levels, impedance etc.

Table 6.3 Tests to be done along with test cases

6.5.5 Case Study #2: Test Cases for Fingerprint Recognition System A fingerprint recognition system is used to allow the entry only to authorized persons. At the entry point, the fingerprint taken is compared with the pre-stored fingerprint database. Here are the various test cases for testing the system:

y Test the system for valid cases (authorized persons). y Test the system for invalid cases (unauthorized persons). y Test the performance by checking the time taken to allow an authorized user and reject an unauthorized user.

y Test accuracy rate by testing for 100 users to calculate percentage of false

recognitions (how many unauthorized persons are allowed entry and how many authorized persons are disallowed—note that allowing unauthorized persons is more dangerous).

y Test the tolerance level of the system by applying ink to the finger of an employee. y Check the ‘overriding’ feature: if an authorized person is disallowed by the system, what is the procedure to override that decision?

y Check the security of the computer in which the fingerprint data is stored by checking the password screen to access the computer/database.

6.6 Document Test Cases The test cases need to be documented as they will be repeatedly used in regression testing. Each test case should contain the following description:

y Test case ID.

Chapter 6

212 y Test case description. y Principle used for test case design (equivalence partitioning, boundary value analysis, use case scenario etc.).

y y y y

Revision history. Author name. Functionality to be tested . Test basis (reference to the section in the work product based on which the test case is designed).

y Environment in which the test case has to be executed (hardware, operating system).

y Test setup. y Pre-conditions (status of the software before giving the test input). y Post-conditions (expected status of the software after executing the test input, including any error messages).

y Test procedure. y Expected results. y Pass/fail criteria. Any changes to be made to the test case design document must be done only through the procedure defined in the configuration management plan. The test cases need to be documented by giving complete details of design and how to execute it. The test case design document has to be kept under configuration control. While documenting the test results, this information has to be appended with the actual results (whether the test passed or failed along with other anomalies if any).

6.7 Execute Test Cases To execute the test case, the exact test procedure has to be worked out. In the case of GUI testing, it may be just some mouse clicks and keyboard entries. In some cases, the exact procedure for interconnecting of the modules, test data to be entered need to be indicated in the test case design document. Based on the test procedure, each test case is executed. The actual results are compared with the expected results. Based on the pass/fail criteria, whether the test has passed or failed is decided.

If you are using testing tools, then execution of the test cases is rather easy. You need to record a test case by actually executing the test case manually. The tool will

Dynamic Testing and Test Case Design Techniques

213 automatically generate a test script. This test script can be executed for executing the test case. This is very useful in regression testing when the same test case is repeatedly executed. In addition to the pass/fail information, the tools also give important information as to when the test was run and how long it took the test case to be executed. This information can be of great use if you want to keep track of the total time of execution of all the test cases. If you are doing manual testing, you need to manually give the test inputs and execute the test cases. If you are using testing tools, you can execute the test cases in unattended mode also. Sometimes, it will not be possible to execute a test case. A test case that cannot be executed because the pre-conditions are not satisfied is called a blocked test case.

6.8 Generate Incident Report/Anomaly Report Any incident that needs to be probed into further is reported by using an incident report (also known as anomaly report). When you execute a test case, if the expected result and the actual result do not match, then it is an incident that needs to be probed into and hence an incident report is generated. Note that incident report may also need to be generated while doing the desk checking of the various work product such as SRS document, design document, source code, test plan etc. The incident report should cover the following details:

y Date on which the incident or anomaly was discovered. y Person/organization who discovered the anomaly (note that the anomaly may be detected by the client organization).

y Development phase in which the anomaly was observed (requirements review, design review, code review, dynamic testing, maintenance).

y y y y y y y

Description of the anomaly. Work product or test case that revealed the problem. Expected result. Actual result. Severity of the anomaly. Priority to be assigned to probe into the anomaly/fix the problem. Present status (to be assigned for fixing, under rectification, rectified, kept on hold, duplicate anomaly as it was pointed out earlier).

y Change history (when the status was changed). y Date of rectification and closure.

Chapter 6

214 Anomaly report or incident report gives a complete record of the anomaly observed in a work product while doing desk checking or while executing a test case.

6.9 Log the Defects For every project, a defect log has to be maintained, preferably by using defect tracking software tool or using Microsoft Excel. Note that defect log is different from the collection of incident reports—a defect may cause multiple incident reports. On the other hand, an incident may or may not be a defect. Each defect is entered into the defect log by the person who discovered the defect. The test manager will study the defect details, assign severity level to it and also assign priority for rectification. For instance, the priority will be high if the testing of the entire product will come to a standstill or if a module testing cannot be done unless that defect is removed. Then the defect removal work is assigned to a developer. Once the developer attends to the defect, then the status of the defect is changed. Sometimes there may be a conflict between the developer and the tester or between the client and the developer as to what is a defect. For example, the location of a GUI button may lead to a conflict. Conflict resolution is a challenge to the project manager.

6.10 Test Documentation Standards Every test engineer has to document his/her work. A number documents need to be generated such as:

y y y y y

Test specifications document/ test design document. Test case design document. Incident report. Test log/defect log. Test closure/completion report.

In this chapter, we studied the information that needs to be documented. It is advisable to follow a standard format for documentation. The IEEE standards on software engineering will be very useful for this purpose. The standards that are of special significance for test engineers are:

y IEEE 829-1998 Standard for test documentation. y IEEE 1028-1997 Standard for software reviews. y IEEE 1044-1993 Standard classification for software anomalies.

Dynamic Testing and Test Case Design Techniques

215 y IEEE 1044.1-1998 IEEE Guide to classification for software anomalies.

6.11 Formal Methods of Testing How well we test software depends on how well we design the test cases. To deliver quality software, we need to follow the process defined in this chapter—identify test objectives, work out the test specifications and the test design, design the test cases, execute the test cases and compare the expected results with the actual results. For all these years, the researchers have been trying to find out whether it is possible to do all these activities automatically. In other words, is it possible to prove that software works correctly by using tools? Proving that the software works correctly involves two things:

y To prove that the program has no errors. y To prove that the program is a true representation of the user requirements. Any program that is written by using a programming language can be converted into a finite state machine—after all, the program changes from one state to another as it keeps executing the statements one after the other. Suppose we can develop a tool that converts the program into its equivalent state transition diagram and checks its correctness. If all the states are correctly defined and how the program moves from one state to another state, is also correctly defined, then we can prove that the program is correct. If some states are undefined or how the program moves from one state to another, is not defined, then the program has errors. Such a tool that checks the correctness of the program is yet to be developed. Even if, this tool is made available, we have to ensure that the program is a true representation of the user requirements. How do we represent the user requirements? If we use a natural language (say, English) to describe the user requirements, there will be lot of vagueness. So, we need to use some formal methods to represent the user requirements such as Unified Modeling Language (UML), data flow diagrams etc. Unfortunately, we are unable to achieve 100% success in representing all the requirements by using formal methods. If and when we achieve it, we can integrate that tool with the tool that checks the correctness of the program, and then the entire software testing process can be automated. Unfortunately, it is still a long way before we achieve formal representation of user requirements and automatically proving the correctness of a program. Till then, software testing continues to be a bit of art, a bit of science, a bit of engineering and a bit of technology—and our jobs as development/testing/quality assurance professionals are safe.

Chapter 6

216

Summary y The process of dynamic testing is as follows: identify test objectives, work out test

specifications and test design, design test cases, execute test cases and compare the expected results with the actual results.

y The item based on which test cases are designed (such as a functional specification, a quality specification or a code segment) is called test basis.

y The test cases are designed by using (a) black box design techniques (b) white box design techniques and (c) experience based design techniques.

y The methods used in black box test case design are (a) equivalence partitioning (b) boundary value analysis (c) state transition analysis (d) decision tables (e) cause effect analysis and (f) use case scenarios.

y White box test case design techniques consider the code at component/unit level, the interconnection of units/components at module level and interconnection of modules at system level.

y Error guessing and exploratory testing are the two methods used in experience based test case design.

y The test case design document contains the details of all the test cases designed. This document has to be kept under configuration control.

y An anomaly report/incident report gives a record of the anomaly noticed during a test case execution or during desk checking.

y Each defect encountered during the test case execution is recorded in a defect log for follow-up by the test manager.

y IEEE standards can be used for the documentation of the various work products generated by using the testing phase.

Review Questions 1. Dynamic analysis tools can be used as: a) Test basis

b) Test case

c) Test script

d) Test identification

2. Test objectives for testing an off-the-shelf software and in-house developed software are the same. a) True

b) False

3. A set of test cases is considered ________ if at least a test case can reveal errors. a) reliable

b) valid

c) trustworthy

d) non-functional

Dynamic Testing and Test Case Design Techniques

217 4. Test cases are designed to test functional requirements only. a) True

b) False

5. To test for both valid and invalid inputs is called, a) Equivalence partitioning

b) State transition testing

c) Boundary value analysis

d) None of the above

6. Testing a test case with multiple data is called data driven testing. a) True

b) False

7. Boundary value testing is applied in: a) Black box testing b) White box testing c) Both black box and white box testing d) None of the above 8. Which of the following standards is for test documentation? a) IEEE 829

b) IEEE 1028

c) IEEE 1044

d) None of the above

9. For a student grading system, design test cases are based on boundary value analysis. a) True

b) False

Detail Questions 1. A program takes degrees as input and gives the sine value as output. What test cases do you use to test the program thoroughly? 2. You need to check whether two phase commit is implemented in a database or not. How do you do it? 3. Write test cases for testing telephone billing software. The software uses the following information about the calls to calculate the billing amount: duration of the call, calling time and date (1/2 rates from 9pm to 6 am and on holidays). 4. Write test cases for sending an SMS message through a mobile phone. 5. Write test cases of library automation software used for issuing/returning books, collecting fines and stock verification at the end of every year. 6. Write test cases for a client/server payroll software. 7. Write test cases for an e-commerce portal that sells only books. 8. Write test cases for an on-line chat program. 9. Write test cases for testing all the functions of MS Word. 10. Write test cases for validation of date. 11. Write test cases for validation of time.

Index

388

Chapter

Managing the Testing Process

7

Chapter Objectives y y y y y y y y y y

Testing process management activities Project planning, budgeting and scheduling Testing team formation Reviews and monitoring of projects Risk management Metrics in testing phase Defect tracking Configuration management Testing process improvement Testing standards and information security testing

To deliver quality product, the project manager has to bring in synergy among various groups. The test manager in turn needs to motivate the test engineers to do a quality job. To ensure that the testing team carries out its job meticulously, the test manager has to carry out a rigorous planning of the testing activities, periodically review the activities, monitor the progress and apply the necessary mid-course corrections. And, after the testing is completed, the test manager has to do a postmortem analysis of the total testing activities and keep improving the testing process so that there is a continuous improvement of quality of products, productivity of people and profitability of the organization. In this chapter, we will study the details of how the testing process has to be managed. Test engineers are advised not to skip this chapter—it is important for every engineer to know what the manager does and also help him in doing his job better.

7.1 Management Commitment Quality assurance program in an organization will succeed only when the top management is committed to quality. Surprisingly, even now, there are many organizations in which there is no separate quality assurance team and testing is done very informally in these organizations. The top management must realize that quality is

Chapter 7

220 the factor that brings in success to the organization. The top management must provide the necessary support for the quality assurance activities by issuing the necessary policy guidelines, and allocating the budget and manpower. To start with, the top management needs to measure the current levels of performance, identify the areas of improvement and establish quality goals. The current levels of performance can be obtained by using the following mechanisms:

y Customer Surveys: If a product has been supplied to a number of customers, the feedback from the customers on the product quality can be obtained.

y Historical Records: For the projects executed/products supplied earlier, the

records of bugs reported by customers after the software is delivered, can be analyzed to calculate the effort/money spent on maintenance and rework. This data reveals the quality of the products supplied earlier.

y The Quality Approaches being undertaken by Competitors: In a competitive

environment, if an organization has to survive and grow, it has to follow the market trends by observing the quality approaches being undertaken by competitors. If all the competitors are obtaining ISO 9000 or CMMI or SPICE certification, the management has to also consider obtaining the certification, otherwise, the organization is no longer competing.

y Industry Standards Benchmarks: Organizations commit themselves to ‘six sigma’

quality, ‘total quality management’ etc. These benchmarks can be also be considered for implementation.

y Own Management Criteria: Sometimes, the management may formulate its own criteria for measuring the quality which can be reflected in: 

large customer base



repeat orders from the same customer



productivity of employees which is reflected by the turnover per employee or profit per employee etc.

After obtaining the current level of performance, the management has to identify the drawbacks of the present process, set quality goals and commit itself to implement a quality management system based on international standards. A number of models have been developed by international bodies which are based on industry best practices. The management can directly adapt these models without trying to reinvent the wheel. The popular models are:

y Capability Maturity Model (CMM)/Capability Maturity Model Integration (CMMI)—staged or continuous representations

y ISO 9001: 2000 y ISO/IEC 12207 y ISO/IEC 15504

Managing the Testing Process

221 Only when the management is committed to quality, the employees can also be made to commit themselves to quality. The management’s commitment to quality is reflected by creating a quality assurance/testing environment. Specifically, the management has to:

y y y y

Issue a policy statement that indicates the management’s commitment. Ensure that the necessary processes have been defined and followed. Provide the necessary infrastructure, manpower and budget for these activities. Provide an environment where there can be a free exchange of ideas and opinions—good communication is a must for quality to succeed.

y Conduct periodic reviews to ensure that the quality processes are being followed and improved continuously.

Quality management systems such as ISO 9001:2000, CMM/CMMI, ISO/IEC 12207, ISO/IEC 15504 have been developed based on the industry best practices. Hence, they can be directly adapted by organizations for implementing their quality systems. Organizations which implement the quality management systems need to have a quality policy. Due to the increasing importance of testing, organizations are now bringing out the Test policy which reflects management’s commitment to testing activities. Test policy can be a part of quality policy.

7.1.1 Organization Structure For any software project to succeed, the project manager has to bring synergy amongst various groups such as domain experts, development experts, marketing experts and quality experts, as shown in Figure 7.1.

Chapter 7

222

Domain Experts

Marketing Experts

Project Manager

Software Development Experts

Quality Experts

Figure 7.1 Synergy of expertise In a traditional organization, a hierarchical structure is followed—typically, the project manager will be in-charge of a project and he/she in turn will control the development and quality teams. This hierarchical structure is not well suited in organizations in which the focus is quality—the project manager may suppress the quality experts’ findings and release the product. A ‘flat’ structure is recommended wherein all individuals work together as a team without any strict hierarchy. Generally, it is recommended that the quality assurance team should report to a top management representative. For instance, in ISO 9001 quality management system, there should be a separate quality assurance team and a Management Representative (MR) will be in-charge of the quality assurance team. The MR will report directly to the top management/CEO. This organization structure indicates that thrust is indeed being given to quality.

7.2 Testing Process Management To deliver quality software, a process-oriented approach is a must. The philosophy behind this approach is very simple [Humphrey 1997]: plan your work, track the work and after the work is completed, do a postmortem analysis to find out what went right and what went wrong. Based on the analysis, improve the process. This is shown in Figure 7.2. If you follow this approach, you can deliver better product, you can plan

Managing the Testing Process

223 better next time and you will be a better person. This approach is applicable to everyone—engineers and managers. In fact, we can use this approach even in our daily lives.

Plan

Track

Post mortem

Better plan, better product and better person

Figure 7.2 Quality improvement process The same philosophy is reflected in PDCA (Plan-Do-Check-Act) model. Here, there are four steps to be followed:

y y y y

Plan: to carry out a detailed planning of your activity. Do: to carry out the activity. Check: to check whether the activity is going as per plan, also take measurements. Act: to take any mid-course corrections, and also to improve the process. Dr. W. Edward Deming, Philip Crosby and Dr. Joseph Juran are considered as the quality pioneers. The foundation for the various models and frameworks has been laid by these pioneers.

7.2.1 Options for Managers One of the important activities of a manager is to study the possible alternatives. If your organization wants a specific software, there are two alternatives: buy or develop. Buying a Commercial Off The Shelf (COTS) software product is a better option, if the software that meets your requirements is readily available from a commercial vendor. Otherwise, the software has to be developed.

COTS Software The advantage of choosing the COTS software option is that the software is readily available and hence it can be bought and can be used without losing time. However, the following aspects need to be considered:

y Selection of the software: This is based on the requirements. A comparison of the requirements and the available features of the COTS software has to be done.

Chapter 7

224 y Vendor selection: The vendor selection is based on a number of factors such as: 

reputation of the vendor



support to be provided for the software



whether local presence (in your country or city) is available



licensing terms and conditions



training to be provided



cost of the software, etc.

y Lack of possibility to make modifications: If you want some special features now or at a later date, the vendor may not be ready to make the modifications to suit your needs. You need to accept the software as it is.

y Need for customization: Some software products such as ERP packages need lot of

customization after installation at your site. Sometimes, the customization cost will be much more than the actual cost of the software. You need to probe into the customization cost before taking a decision.

y Testing the COTS software: The software has to be tested to ensure that it meets all your functional and non-functional needs. The detailed testing process has to be worked out. Testing the COTS software also needs to be done using a rigorous testing process. The process we discuss in this chapter can be suitably adapted for COTS software testing.

Software Development If COTS software is not available, you need to consider the option of development. The development can be done by using in-house development team or development can be sub-contracted. If the development is sub-contracted, you need to carry out the testing to ensure that the software is as per your requirements. If the development is done inhouse, you need to have a separate testing team to test the software. You also have an option to sub-contract the testing activity. Suppose the top management has given you the responsibility to make a payroll software package operational in your organization. The software has to maintain the information about the salaries of the employees and also generate the necessary reports for submission to various government organizations (such as income tax department, provident fund organization etc.) every month. You have to consider the options: buy COTS software or get it developed. If you have to buy COTS software, you have to get the details of the payroll software packages available in the market and compare their specifications with your requirements. You have to shortlist a few packages that meet your requirements and then carry out a detailed study of each package—the detailed specifications, reputation of the vendor, support provided by the vendor, the training to be given by the vendor and the

Managing the Testing Process

225 cost. After selecting a particular vendor’s package, you need to test the software. This testing basically involves checking whether it meets your needs or not and whether it is reliable or not. You cannot ask for any modifications to be done by the vendor as it is a generic package being sold by the vendor. On the other hand, if you decide to get it developed, you can either get it developed by the in-house development team or subcontract the development work. In either case, testing is your responsibility. Here, you need to test the software, and then give the feedback to the development team so that the bugs can be rectified and finally suitable software is delivered to you.

7.2.2 Testing Process Management Activities A well-defined testing process is fundamental to develop the quality software. Generally, in ISO 9000/CMM/CMMI certified organizations, an organization-wide testing process is developed. Based on this process, a process needs to be obtained specific to a project. This process is fine-tuned based on the software development life cycle model being used for the project. The process should also define the objectives of testing, methodologies to be used, infrastructure and tools required, method of testing, work products of the process and the reviews to be conducted during the testing phase. The process should also identify the training needs of the people. The manager in-charge of the testing process has to carry out the following activities:

y y y y y

Planning the testing process, budgeting and scheduling.

y y y y y y

Managing the risks.

Aligning organization-wide testing process to the project. Forming the testing team. Providing the necessary infrastructure. Executing the test plan and regularly reviewing the work and monitoring the progress. Obtaining the test metrics. Tracking the defects. Carrying out configuration management. Doing a postmortem analysis, after the project is completed. Based on the postmortem analysis, improving the testing process.

We will discuss these activities in the following sections. The test manager has to meticulously plan all activities to be done during the testing phase and prepare a test plan. The test plan is used as the reference throughout the testing phase.

Chapter 7

226

7.3 Planning, Budgeting and Scheduling the Testing Phase Figure 7.3 shows the activities in the planning phase. The testing requirements need to be studied in detail and the objectives of testing, criteria for completion of testing need to be worked out. Once the testing requirements are worked out and the detailed tests are designed, the effort/budget/time required for the testing phase can be estimated. Testing requirements and test design activities should be carried out, in parallel, while the development work is going on. The effort, time and budget required for testing are worked out and based on the perceived risks (project risks and product risks), the effort estimation is fine-tuned.

SRS Document

Study Testing Requirements

Test Design

Estimation of 1. Effort 2. Time 3. Budget

Risk Analysis 1. Process Risk 2. Product Risk

Prepare Test Plan Document

Figure 7.3 Test Planning Process

Managing the Testing Process

227 During the project proposal stage, already the project manager would have given the total time frame that includes the time for testing. It is advisable that test planning activity be done separately and these inputs be taken into consideration while preparing the project proposal.

y Test planning involves identification of the following: 

What are the testing objectives and the scope of testing?



How to align the overall process to the project?



What are the criteria for completion of testing?



What types of testing have to be carried out?



Which testing tools have to be used?



What is the infrastructure required for testing?



What are the work products to be generated during the testing phase?



How much effort is involved in testing?



How much money and time are required?



What are the skill sets required for the testing professionals?



What is the team structure?



What are the risks associated with the project and how to manage risk?

y Testing Objectives: For every project, the test manager has to clearly identify the objectives. The objectives can be:  









To find the defects and report to the QA/development team. To fix the defects as well as suggest how to improve the software (for example, suggestions on the user interface). To carry out a review of the source code for completeness, adherence to coding standards/guidelines. To carry out non-functional testing such as for portability, security, safety, conformance to international standards, accessibility, internationalization etc. To check whether the software meets the functional and interface requirements in case of COTS software. If acceptance testing by the customer is assigned as a task to the testing team, acceptance testing is also an objective.

y Criteria for completion of testing: It is very important to identify the criteria for

completion of testing at early stages of development. Remember that software testing is never complete—at some point, we need to say, enough testing, let’s deliver. But this decision has to be taken based on some criteria which need to be identified during the planning stage.

Chapter 7

228 Identification of testing objectives and criteria for completion of testing are the two important aspects that need to be worked out in detail so that a realistic estimation of effort, budget and time can be made. Based on the testing objectives and criteria for completion of testing, you will be able to approximately estimate the effort/cost required for testing in person months. This effort estimate needs to be adjusted keeping in view the risks associated with the project. As discussed earlier, you need to take into consideration the product risks and the project/process risks. For example, if you perceive that some people are likely to leave in the middle of the project, then you would like to put additional people from the beginning of the project. In such a case, you need to add the additional manpower cost to the previously estimated cost. If you perceive a technology risk and want to appoint a consultant to provide guidance, you need to add the consultant’s fee to the estimates. Please note that effort/time/budget estimation is not a very easy task. The two approaches generally followed for effort estimation are:

y Experience-based. y Metrics based. In majority of organizations, effort estimation is based on past experience. Experts who worked on similar projects earlier estimate the effort, purely based on their ‘gut’ feelings. Since they are experts, their estimates turn out to be reasonably accurate. A more scientific approach is to use metrics. If your organization kept records of projects executed earlier, then it is very easy to use the metrics-based approach. For instance, one metric that can be used is the total number of lines of code. Suppose in an earlier project, there were 100,000 lines of code and the testing could be completed in 4 months by 5 test engineers. So, each test engineer could complete testing of 5,000 lines of code per month. If the present project is also of similar nature, and the total code size is 200,000, you can reasonably estimate that 40 person months of effort is required to do the testing. Of course, remember that the two projects must be of the same type (same application domain, same programming language, same set of tools are used etc.). On similar lines, the effort can be estimated by using the following metrics:

y y y y

Size in function points. Size in number of classes and subclasses, if object-oriented design is done. Size in number of tables, forms and reports in a database application. Size in number of pages of a web site.

Once the effort in person months is estimated, the budget can be calculated based on the cost of one person month. If one person month cost is taken as say US$ 4000, then the total budget will be: 4000 * number of person months

The schedule for the testing phase is generally dependent on the overall timeframe of the project. Generally 30-40% of the total time is required for testing phase but that much

Managing the Testing Process

229 time is never allocated. Unfortunately, the test engineers always have to work with very tight schedules. Two methods based on which effort estimation is done are: (a) experiencebased and (b) metrics-based. Generally, experts in an organization do the estimation based on their past experience. But, it is advisable to use metrics and historical data to do the estimation. Effort estimation is a very difficult task. Even then, it is advisable to develop a process for estimation and keep refining that process rather than relying on one’s gut feeling.

7.3.1 Test Plan The test plan, containing the details of the testing process, has to be prepared during the planning stage itself. This plan should contain all the details of required resources, the testing approaches to be followed, the testing methodologies, the test cases etc. This section will address these aspects of the test plan. Before taking up the testing, a test plan has to be prepared. It is preferable if the test plan is prepared during the overall project planning stage itself and revised just before the testing process starts (as the issues involved in testing will be clearer after the coding is completed). A template for the test plan is given in Form 7.1. Test Plan Project name: ______________________________________________________________________ Estimated start date for testing: ________________________________________________________ Estimated end date for testing: _________________________________________________________ Actual start date for testing: ___________________________________________________________ Actual end date for testing: ___________________________________________________________ Estimated effort in person hours/person months for testing: _________________________________________________________________ Estimated budget for testing: __________________________________________________________ Work products based on which tests are designed (SRS document, design document etc.): __________________________________________ Test set-up (including the hardware, software environment), other peripherals required, any special tools required and any test equipment required (such as oscilloscopes etc.): ___________________________________________________ Test personnel and their responsibility (Test manager, test leaders and test engineers): ________________________________________________

Chapter 7

230 Training to be provided to the test engineers: ____________________________________________ Types of testing to be carried out (functional, structural, alpha, beta, Gorilla, usability, field trial, performance, stress, accessibility, conformance etc.): _______________________________________ Checklists to be used for static testing and dynamic testing: _________________________________ Testing tools to be used: _____________________________________________________________ Work products to be generated during the testing phase (test case design document, system test results document, code coverage test results, acceptance test results document, testing completion document): ____________________________ For each type of testing, the test cases have to be specified: _________________________________ Defect report format and defect tracking mechanism: ______________________________________ Format for defect reporting (incident logging) S. No.: __________________________________________________________________________ Defect found: _____________________________________________________________________ Type of defect: ____________________________________________________________________ Classification of the defect (critical, major, minor): _______________________________________ Whether the defect is removed: _____________________________________________________ Time for removal of defect: _________________________________________________________ Stage when the defect was injected: ___________________________________________________ Stage when the defect was removed: __________________________________________________ Details of configuration management: work products to be kept under configuration control, the configuration management team: ____________________________________________________ Metrics to be collected: ______________________________________________________________

Form 7.1 Test plan format The test plan has to be kept under configuration control. Any changes to the test plan have to be done systematically as per the configuration management plan.

7.4 Alignment of the Process to the Project In organizations with ISO 9001/CMM/CMMI certification, organization-wide processes will be defined. These are generic process definitions applicable for all the projects. Based on this processes, the processes specifically required for the project on hand need to be

Managing the Testing Process

231 worked out. Alignment of the process to the project involves fine-tuning the testing process for the project based on the overall testing process definitions. This alignment has to be done keeping in view the following aspects:

y y y y

Whether the software is COTS software or custom-built. Whether the software is developed in-house or through a subcontractor. The software development life cycle chosen. Whether procedural languages (like C) have been used or object-oriented languages have been used.

y The application domain (ERP, CRM, telecom software, networking software, embedded software etc.).

y Whether development/testing models such as extreme programming and buddy testing are to be used.

y Whether testing tools have to be used. If organization-wide processes are not defined, a process specific to the project on hand can be developed. In organizations with ISO 9001/CMM/CMMI certification, organization-wide process for testing will be defined, which is a generic process definition. Alignment of the process for the project involves fine-tuning of this process to the project on hand keeping in view the project’s specific requirements.

7.5 Team Formation The quality of testing and as a result the quality of the product depends solely on the people employed for testing. The manager may take excellent individuals with strong technical background, but still the team may not perform well. Even if, all the players in a team are excellent, they may not be match-winners—be it cricket or soccer or testing. For the team to succeed, each individual in the team has to play the different ‘roles’ needed to achieve the goal. The Team Roles Concept was formulated by Dr. Meredith Belben can be applied effectively to build good teams that achieve the goals. According to this concept, people that can play different roles are required in the team. These roles can be divided into three categories:

y Action-oriented roles: Shaper, implementer, completer/finisher. y People-oriented roles: Coordinator, team worker, resource investigator. y Cerebral roles: Monitor, evaluator, specialist. Broadly, the roles are action-oriented or social-oriented or thinking-oriented. This simple concept needs to be kept in mind by the manager while forming a team. If the team has all ‘thinkers’ and no ‘implementers’, then the work will never be completed. You can get more details about team role concept from www.belbin.com.

Chapter 7

232 Dr. Meredith Belben’s team roles concept can be used effectively to form the winning teams—whether in sports or in software testing. The team should have people who can play different roles which are action-oriented, peopleoriented and thinking-oriented. If the team consists of individuals who all excel in only one aspect, they cannot produce results.

Another model that is widely used for forming teams is based on Margerison-McCann Team Management System (www.tms.com.au). According to this model, for the team to succeed, persons with different strengths are required. However, the personal work preferences differ. Only when such people with a common characteristic of ‘linking with others’ come together, the team will succeed. According to this model, every team members should be a strong team player. In addition, based on the type of work to be done and personal work preferences of individuals, the team should be selected. There are 8 types of profiles defined. Every individual will belong to one of the following profiles:

y y y y y y y y

Reporter-Adviser Creator-Innovator Explorer-Promoter Assessor-Developer Thruster-Organizer Concluder-Producer Controller-Inspector Upholder-Maintainer

Based on a simple set of questionnaire, it is easy to assess and find out to which category a person belongs. Team formation can be done by assessing the individuals’ preferences and matching the team requirements with these preferences. For a testing team to succeed, the test engineers need to have the following skills:

y y y y y y y y y y

Good understanding of the application domain. Excellent knowledge of the software quality assurance principles. Thorough knowledge of software testing. Exposure to testing tools. Experience in designing test cases. Good oral and written communication skills. Good listening skills. Good inter-personal skills. An eye for detail. Excellent analytical abilities.

Managing the Testing Process

233 Every organization devises its own way of recruiting people. In the job interviews for Microsoft [Cusumano 1999], some of the questions asked are: “how much water flows in the river Mississippi?”, “how many gas stations are there in the country?” Note that the exact answers to the questions are not really important, what is important is how you approach a problem to find a solution. When you are trying for a job, you need to tolerate all the idiosyncrasies of the HR managers and CEOs.

The skills required for test leaders/test managers are:

y y y y

Ability to form high-performance teams.

y y y y

Mentoring of the test engineers.

Ability to identify strengths of the individuals. Good written and oral communication skills. Ability to resolve conflicts (that arise between testing team and development team; development team and client). Motivate and inspire the test engineers to work with strict deadlines. Leadership qualities. Facilitate a group to achieve the goals.

The test manager’s responsibilities are:

y y y y y y y y y y y y

Plan the testing process. Form the team and provide the training to the team members. Ensure that the necessary infrastructure and tools are available. Conduct periodic reviews and manage risk. Monitor the progress and communicate the status to all persons concerned. Coordinate with different groups. Resolve any conflicts that arise between individuals and groups. Identify and collect metrics data. Configuration management. Ensure that all the testing objectives are met. Keep improving the testing process. Ensure that each test engineer fulfills his/her responsibilities.

The test engineers’ responsibilities are:

y Study the work products and work out the test specifications including the types of tests to be done.

y Carry out test design.

Chapter 7

234 y y y y

Test case design and generate test scripts. Document test cases. Test case execution and incident reporting. Report the progress to the manager/leader. The test manager should work out a ‘job description’ for each test engineer. The job description gives the details of the role and responsibilities of the test engineer. In addition to collective responsibilities, it is equally important to have individual responsibilities.

7.6 Infrastructure The test manager has to ensure that the necessary infrastructure is available to the testing team. The infrastructure includes:

y y y y y y y

The working area with provision for refreshments and some recreation. Computers and networking. Communication facilities (telephone, Internet connectivity). Software testing tools. All process related documents. Work products of the project. Special testing equipment such as protocol analyzers, oscilloscopes, spectrum analyzers etc. (for embedded software testing).

7.6.1 Testing Tools Test engineers’ productivity can be increased many fold if testing tools are used. In most of the projects, enough time is not allocated to the testing phase. As a result, testing is done in a hurry and buggy software is delivered to the customer. If testing tools are used, the testing time and budget can be reduced considerably and quality of the product can be improved.

Managing the Testing Process

235 Testing Tools

Functional Testing Tools

Performance/Load Testing Tools

Profilers

Source Code Testing Tools

Time Profilers

Test Management Tools

Support Tools

Specialized Testing Tools

Metrics Calculations

Bug Tracking Tools

Configuration Management Tools

Integrated Test Management Tools

Documentation Tools

Requirements Tracking Tools

Embedded Real-Time Testing Tools

Protocol Test Tools

Conformance Testing Tools

Figure 7.4 Classification of Testing Tools A number of testing tools are available—both open source tools and commercial tools. The classification of the tools is shown in Figure 7.4. We will discuss the details of the testing tools in Chapter 8.

7.7 Reviewing, Monitoring and Risk Management It is naïve on the part of the manager to expect everything to go as per the test plan. Invariably nothing will go as per the plan. The manager has to constantly review the work of the individual test engineers, monitor the overall progress and take corrective action. It is likely that there is a threat to the project execution due to the risks. Risk control has to be done during the testing phase. A very simple and effective tool for reviewing the progress of the project is through Gantt chart, also known as bar chart shown in Figure 7.5. In this chart, the important activities are shown by bars. Each activity is represented by a bar along with the start date and completion date. This bar chart is prepared at the time of planning and periodically, it is checked whether the activities are going on as per the plan. Another bar

Chapter 7

236 with the actual time taken can also be drawn for each activity, in parallel to the present bars. This chart is generally used by the top management for reviewing the activities.

Testing Tools Training

Acceptance Testing

Test Execution

Test Case Design

Test Specification and Design

T0

T0+1m

T0+2m

T0+3m

T0+4m

T0+5m

Figure 7.5 Gantt (bar) chart The IEEE Standard 1029-1997 Standard for software reviews is the important reference based on which reviews can be carried out.

If the project manager also wants to keep track of the estimated effort for each activity along with the time frame, the cost-schedule-milestone graph shown in Figure 7.6 is used. In this figure, X-axis represents the time and Y-axis represents effort in person months or cost in dollars. On the graph, the various milestones are indicated such as test specifications, test case design, test execution etc. At the time of planning, the estimated values are plotted on the graph and while executing the testing phase, the actual values are plotted. So, periodically the manager will know whether he is ‘earning’ or ‘losing’— in terms of cost and time.

Managing the Testing Process

237

Field Trial

Effort

Test Report

Actual Execution of Tests

Test Specifications

Test Design

Planned

Time (month)

Figure 7.6 Cost–Schedule–Milestone chart

7.7.1 Risk Management As discussed in earlier chapters, we need to take risks if we want to do business. But if we take blind risks or if we do not try to control the effect of risk, then the project is doomed. We also studied that ‘risk-based testing’ is one way of working out the testing strategies. Fundamentally, the project manager has to assess the risk associated with releasing untested software to the client. During the planning stage, the various product risks and project risks are identified. Suppose ‘persons leaving in the middle of the project’ is identified as a risk item. If really a few people leave the project, the manager has two options: (a) transfer some test engineers from other divisions or (b) take test engineers on a temporary basis. If risk management is done well, the manager will not worry, because he would have already put some additional people on the project anticipating the risk.

7.7.2 Test Reports The manager has to release periodically status reports indicating the following:

y y y y y

Types of tests carried out and pending tests. Percentage of requirements covered by testing so far. Code coverage achieved by testing so far. How much testing is still left out. Total effort, time and budget already spent.

Chapter 7

238 y Whether testing will be completed as per schedule or likely to be delayed. y Any issues to be resolved. At the end of the testing phase, a test report has to be generated by giving the following details:

y y y y y y y

Results of various tests carried out. What criteria have been used for declaring that testing is complete? Whether the software is ready for delivery or not? Suggestions for improving the product quality. Actual time, effort and money spent on testing. Various metrics collected. Statistical analysis of the defects (defect density, defects removed per hour) etc. Periodic reviewing of the testing activities is a must. Reviews help in ensuring that testing is being carried out as per the plan and also to take any mid-course corrections if required. Gantt chart and Cost-Schedule-Milestone are the two charts that are useful to managers for reviewing the project activities.

7.8 Metrics To effectively manage the testing process, the manager has to identify the metrics and obtain the measurements. Some important metrics are as follows:

y Effort (in person months), time and budget estimated and the actual values after completion of the testing phase. The reasons for the difference between estimates and actual values need to be studied—is it a problem with estimation or execution?

y The total number of lines of code, number of functions, number of classes in object-

oriented design, number of tables and forms in a database, number of web pages in a web site development etc. (also called Project complexity metrics).

y Total number of defects found and defects found in each category (for example, critical/major/minor defects).

y Root cause of defects (how many defects were due to problems in requirements, problems in design, problems in coding etc.).

y Severity of defects and percentage of defects in each severity level. y Defect removal efficiency (how many defects were removed per hour). This gives an indication of the productivity of the test engineers.

y Total number of test cases executed and the percentage of test cases that detected defects.

y Reliability metrics (MTBF, MTTR and MTTF).

Managing the Testing Process

239 y Testing effort/time/budget as a percentage of total project effort/time/budget. y Cost of quality (failure cost, appraisal cost, preventive cost). The metrics have to be estimated during planning stage and measured during the test execution stage. The metrics help in better monitoring of the progress and also in improving the testing process..

7.8.1 Software Reliability A number of reliability analysis models are available to predict the reliability of the software, in terms of the number of defects still remaining in the software [Wood 1996]. This metric is very useful to decide when to deliver the product or when to release the product into the market, particularly when the software is highly complex. The software reliability models generally follow one of the trends as shown in Figure 7.7. Fig 7.7(a) shows the concave model’ and Figure 7.7(b) shows ‘S-shaped model’. The assumption made in arriving at these models is that during initial stages, testing is not as efficient as at later stages (not necessarily a true assumption) and hence there is an increase in the rate of defect detection. In Figure 7.7(c), the two-stage model curve is shown, this model is applicable when significant functionality is added to the software during the testing process (which is normally the case).

No. of Defects

No. of Defects

No. of Defects

Test time

Test time

Test time

(a) Concave

(b) S-Shaped

(c) 2 -stage

Figure 7.7 Software reliability curves The software reliability models are still at a theoretical level and studies are being carried out at academic institutions. Very few organizations use them in practice for obtaining the reliability figures. The most important reliability figures are MTBF (Mean Time Between Failures), MTTR (Mean Time To Repair) and MTTF (Mean Time To Fail). The relation among these three values are shown in Figure 7.8.

Chapter 7

240

Failure

Repair

Failure MTTF

MTTR MTBF Figure 7.8 Relation among MTBF, MTTR & MTTF

While carrying out the testing process, the values of MTBF, MTTR and MTTF need to be recorded and if these values are within the limits only, the product can be considered as ‘ready for release’. The software reliability models have to be developed by using the defect analysis during the testing phase and the maintenance phase. Based on these models, the reliability estimation models can be evolved in the organization. Organizations, which would like to improve their process based on quantitative measures, need to apply the reliability models as a part of their quality improvement process. The parameter Mean Time Between Failures (MTBF) is used to measure the reliability of the software product. However, to measure this product, the testing period has to be very large. Alternatively, statistical techniques are used to calculate the MTBF value.

7.9 Defect Tracking While executing the testing phase, every defect has to be recorded in a defect log (also known as incident report). The test manger has to define a process for defect identification to disposal of the defect. This process consists of the following steps:

y Recognition of a defect: For a particular test case, if there is a difference between

the expected result and actual result, then it is an anomaly or defect and it has to be recorded. This defect has to be recorded in a defect log. The test engineers can only enter the defect into the log, they cannot remove data from the log.

y Investigation as to why the defect occurred: The root cause of the defect has to be

identified in association with the developer, if required. The test manager has to study the defect log, find out the severity of the defect, assign priority to its removal (high, low, can be ignored for now) and identify the person who will remove the defect.

Managing the Testing Process

241 y Defect removal: The defect has to be removed by the developer. When the defect is

removed, the test manager again needs to ensure that confirmation testing and regression testing are done and confirm that the defect has been really removed and that defect removal did not introduce any other new defect. Then the test manager can declare that the particular defect is disposed off.

7.9.1 Classification of Defects All defects are not of the same type. Some defects may result in system crash. Some defects may be very minor (such as a spelling mistake in an error message). So, the defects need to be classified based on its impact on the functionality of the software. The defects can be classified as:

y Critical y Major y Minor Critical defects result in a system crash, or the application may not be useable at all if this type of defect occurs. In a time critical application, if the timing is not within the limits or if the response time is not within limits; then also the defect can be termed as critical. Inconsistent performance of the application is also a critical defect. Major defects will not lead to a system crash, but may result in some portions of the application difficult to use, such as difficult navigation of the menus. Minor defects can be tolerated, such as lack of help for some functionality, a spelling mistake in an error message, incorrect ordering of control buttons in a GUI etc. While recording the defects, the severity of the defects also needs to be noted down. If a critical defect is present in the code, then the software is not ready for delivery. Only a few major defects are allowed. Similarly a threshold needs to be kept on the number of minor defects. As the threshold depends on the application, the project manager/test team member has to use his judgment in fixing the thresholds on major and minor defects. Another way of classification of defects is based on functional/non-functional requirements. Here, the categories can be:

y y y y y y y

Functional defect Non-functional defect Security-related Safety-related External interface related Conformance related Portability related

Chapter 7

242 y Usability related y Testability related For each category, severity can be assigned (the severity levels can be 3 or 5). After analyzing the test report, the project manager needs to decide whether the software is ready for delivery or not. During testing, each defect found has to be classified as critical, major or minor. A threshold has to be fixed on each category of defects and if the number of defects in that category is more than the threshold, the software is not ‘good enough’ for delivery.

7.10 Configuration Management Configuration management is an activity that spans across all phases of development. However, it is during the testing phase, many defects are detected and as a result, the request for change is initiated. In general, as shown in Figure 7.9, when a product is released or when a mature product is given to the testing team for testing, a change request may be initiated due to:

y Preventive maintenance: Anticipating a problem, change is being made so that the software will not fail in future.

y Adaptive maintenance: Due to change in requirements, or due to the need to enhance the features of the software, a change is being made.

y Corrective maintenance: A problem has been encountered in the software and a change is being made to correct it.

Managing the Testing Process

243 Preventive

Product

maintenance

release

Adaptive

Change

maintenance

request

Corrective maintenance

Change request analysis [impact Analysis] Work product

Release planning

Change implementation Change release System testing

Figure 7.9 Change management Whenever a change request is received, a detailed impact analysis is made and then it is decided whether the proposed change should be implemented or not. If it is decided to implement the change, the necessary changes are made not only in the code, but in all the work products and the software is thoroughly tested and then released. Before accepting a change, a thorough analysis of the impact of the change has to be done, as shown in Figure 7.10. The impact of the change on both functional requirements and non-functional requirements has to be studied by the technical team and the impact and the budget, time and effort have to be studied by the management. If the impact is not very high, or if the impact is high but the client is ready to foot the bill, then the proposed change can be accepted.

Chapter 7

244 Change request Budget

Functional requirements

Impact analysis (Management)

Time

Impact analysis (Technical)

Performance requirements

Other non-functional requirements

Manpower

Decision on Change request acceptance

Change Implementation

Figure 7.10 Change Impact analysis In many projects, the managers innocently accept a change proposed by the customer, when the customer tells that it is a ‘minor’ change. Without studying the impact (or the damage to be caused by the change), if the proposed change is accepted, and later on if the change turns out to be major—it is going to affect the profits of the organization. Particularly, when changes in performance parameters are requested by the client, the development team needs to be extremely cautious. During the testing phase and after releasing the product, a number of change requests will be made due to (a) preventive maintenance (b) corrective maintenance or (c) adaptive maintenance. Whenever a change is proposed, the impact of the change (or the damage likely to be caused by the change) on the functionality, performance and other non-functional requirements as well as effort, time and budget have to be analyzed.

Managing the Testing Process

245

7.11 Test Closure and Process Improvement When the testing phase is completed, the manager and all the team members would gain lot of experience from the project. The experience helps in improving the testing process further. As shown in Figure 7.11, for every project, we define a process and set quality goals. We carry out the testing activities as per the plan using the process definitions. Then we need to compare the quality goals with the actual results. If there is no match, we need to study where there is a scope for improvement. This postmortem analysis helps in finding our weaknesses so that we can work on how to overcome those weaknesses.

Define a process

Set quality goals

Plan testing activities

Track the testing activities

Compare goals and results

Process refinement

Figure 7.11 Improving the testing process Using the metrics, we can find out the deficiencies in our processes and people, and try to rectify these deficiencies.

y If the testing phase could not be completed as per the estimated time frame or budget or effort, the manager has to find out the reasons. Is it due to the problems in estimation? If so, the estimation process has to be improved. Otherwise, if there are problems in execution, then the execution process needs improvement.

y If it is realized that more defects are due to improper specifications/design, the requirements engineering/design process has to be improved.

y If more defects were introduced in coding stage, the programmers need to be trained on programming.

y If there are too many defects being reported by the customer, then the testing phase is not executed well and hence the test manager needs to be trained.

y If the defect removal density is low, test engineers need training or more sophisticated tools need to be introduced.

Continuous improvement of the testing process should be the aim of the management. We need to appreciate that our success today is based on our past experience. Our experience is based on our past failures. And, we need to learn from failures—in fact, we can learn more from failures than successes.

Chapter 7

246

7.11.1 Software Testing Maturity Model (SW-TMM) Software Testing Maturity Model (SW-TMM) was developed Dr. Ilene Burnstein of Illinois Institute of Technology and her associates. This model is similar to CMM/CMMI and can be used effectively by software testing organizations to improve their testing processes systematically, stage by stage. In SW-TMM, there are 5 levels: Level 1: Initial Level 2: Phase definition Level 3: Integration Level 4: Management and measurement Level 5: Optimization/defect prevention and quality control

y Level 1 organizations are characterized by: 

Chaotic testing activities.



Testing is done on ad hoc basis.



Testing is done just to prove that software works.



Testing and debugging are not differentiated.



Staff not trained in intricacies of testing.

y Level 2 organizations are characterized by: 

Testing is recognized as a separate phase after construction /implementation.



Testing is considered as separate from debugging.



Basic testing methods are followed.



Testing is done to show that software meets the requirements.

y Level 3 organizations are characterized by: 

Management considers testing as a professional activity and gives the necessary support and thrust to testing phase.



Staff is trained thoroughly on testing process.



Formal testing processes for planning, reviewing and monitoring are in place.



Tools are used for testing.



Testing objectives are defined on the basis of software requirements.

y Level 4 organizations are characterized by: 

Metrics are used for managing the testing process (testing process is measured and quantified).



Quality parameters such as reliability, usability, security etc. are tested.



Test cases are designed, documented and reused for regression testing.



Defects are tracked.

Managing the Testing Process

247 y Level 5 organizations are characterized by: 

Well defined and managed testing process.



Testing effort and cost are planned and monitored.



Testing tools are evaluated, selected and introduced based on a defined process.



Testing process is continuously improved.

SW-TMM is a simple model that can be followed by every organization even if it is not following CMM/CMMI. To start with, an organization has to assess itself at which level it is at present based on the current process definitions. Then, it has to implement the various processes to achieve Level 5. Software Testing Maturity Model (SW-TMM) can be used to assess the maturity level of an organization in testing. Five maturity levels are defined— level 1 organizations have no process definitions for testing, level 5 organizations have well defined and managed testing process. Even if your organization has no CMM/CMMI certification, you can adapt SWTMM for your testing activities.

7.12 Information Security Nowadays, test engineers and test managers have to give lot of thrust to information security. Using the software products and services, lot of information is stored in the computers and this information has to be kept secure. Security of information stored in the computers has become a major concern. The statistics on cyber-crime is very unpleasant:

y Almost every business or government organization is affected by attacks on information systems one time or the other.

y More than 80% of the people who shop online are concerned about the credit card fraud.

y Employees and ex-employees cause nearly 50% of the security breaches. y Viruses such as Melissa, I Love You cause millions of dollars of information loss across the world every year.

Information systems (paper file racks, computers, networks and people) are prone to attacks—the attacks may be natural (flood, earthquake, tornado, cyclone etc.) or manmade (fire, theft, vandalism, cyber-attack). We need to identify the important ‘information assets’ of the organization and ensure that these assets are not prone to attacks. If a system is prone to attacks, it is said to be vulnerable to attacks. If a system is vulnerable to certain type of attack, we need to put some security controls so that we can reduce the risk associated with the damage that can be caused by that attack. For

Chapter 7

248 instance, if a particular office is prone to cyclone attacks, it is better to maintain another data center that will have an exact replica of this office information assets. If a particular location is prone to some intruders getting into the premises, then access control mechanisms such as entry through card punching, smart card, fingerprint etc. can be implemented. Keeping in view the importance of providing security to the information systems, a number of international standardization bodies and government organizations took initiatives to develop standards for developing secure information systems. All these initiatives aim to provide a framework for ‘security assurance’ similar to ‘software quality assurance’. Security assurance involves:

y To find out the flaws in the present security system by carrying out a rigorous vulnerability analysis, also known as penetration testing.

y To define a security policy that indicates the management’s commitment to the security aspects of information system, and ensure that each and every employee is also committed to the security policy.

y To define security procedures and guidelines that need to be followed across the

organization. These procedures include user management and password protection, anti-virus software usage and updating, backup and restoration procedure, user training on security aspects, auditing of the security system, and disaster recovery and business continuity mechanism.

A number of laws have been enacted by different governments on cyber laws that address computer crimes, such as:

y The US government released an actUniting and Strengthening America

by providing Appropriate Tools required to Intercept and Obstruct Terrorism Act (abbreviated USA Patriot Act) after the 9/11 attacks in 2001.

y Indian Information Technology Act 2000 was released by Government of India in the year 2000. This act addresses various aspects of electronic documents, digital signatures and punishments for cyber-crimes.

y Health Insurance Portability and Accountability Act (HIPAA) was released by

department of Health and Human Services, USA in the year 2003. This act is for protection of health information of the patients so that the individual privacy is not compromised. Organizations involved in health care services such as hospitals, health care clearing houses, health insurance agencies etc. have to ensure that their information systems are HIPAA-compliant. HIPAA addresses five areas for protection of information: 

administrative controls



physical controls



technical controls



organizational requirements and

Managing the Testing Process

249 

policy, procedures, and documentation requirements.

y Graham-Leach-Billey Financial Services Modernization Act (GLBA) was released in

the year 1999 in USA. This act addresses protection of customer’s private information by financial institutions. As customer information related to financial matters is handled by multiple agencies such as banks, credit card companies, insurance agencies etc., all these agencies need to follow the guidelines specified in the document Inter-agency Guidelines establishing Standards for Safeguarding Customer Information. Most of the financial/banking agencies subcontract their software development/testing work. Hence, the subcontractors need to be familiar with the GLBA and implement the necessary security controls to comply with the requirements of GLBA.

To provide security controls to information systems, we need to put in place Information Security Management Systems (ISMS). The ISMS can be developed based on international standards such as:

y In Common Criteria (ISO 15408) standard, a product is evaluated based on seven evaluation levels. These levels are: 

EAL1: The product is tested for it functionality.



EAL2: The product is tested structurally.



EAL3: The product is methodically tested and checked.



EAL4: The product is methodically designed, tested and reviewed.



EAL5: The product is semi-formally designed and tested.



EAL6: The product is based on semi-formally verified design and tested.



EAL7: The product is based on formally verified design and tested.

y In BS 7799 (ISO 17799) standard, four types of security assurance areas are addressed. These are: 

Organizational assurance (internal to the organization).



Service provider assurance (external third party suppliers).



Trading partner assurance (business partners).

y Information processing facilities assurance (including product suppliers). In this standard, for testing the products, the security evaluation levels of CC can be made use of—levels 1 to 7. Level 1 means that the product cannot be trusted and level 7 means that the product is highly trustworthy. This standard also gives details of security policy, how to classify information assets, various security controls to be put in place such as personnel security, physical security, environmental security, access control, business continuity management etc.

y OCTAVE (Operational Critical Threat, Asset and Vulnerability Evaluation) approach developed by Software Engineering Institute. This approach is based on a

Chapter 7

250 simple philosophy to identify the critical information assets of your organization, study the threats, identify the vulnerabilities, and develop a security strategy and plan. Implement that plan and audit periodically to check whether the implementation is good or not, and take corrective actions if required. OCTAVE-S is a scaled down version of OCTAVE that can be used by small organizations.

y SSE-CMM (System Security Engineering Capability Maturity Model) is also

developed by Software Engineering Institute. In SSE-CMM has 6 maturity levels— Level 0 organizations have no security processes and everything is done informally and Level 5 organizations have excellent security processes which are continuously improved and hence they are highly trustworthy. In this model, there are 21 process areas that need to be implemented to achieve level 5.

y Committee of Sponsoring Organizations (COSO) of Treadway Commission

released a report in 1992 which specifies the various internal controls to be incorporated to provide security to information/software systems. As per the principles suggested by COSO, the existing security environment has to be evaluated, risk assessment has to be done, vulnerable points are identified, security controls are put in place and it has to be ensured that the control are sufficient to meet the security requirements. For example, consider a bank transaction. The various vulnerable points are transaction origination, transaction entry, transaction process, database control, and transaction results. At every point, it has to be checked whether the controls are adequate and effective. COSO equivalent in Canada is CoCo (Criteria of Control).

y Control Objectives for Information and related Technologies (CobiT) developed by

IT Governance Institute (ITGI) gives a framework which can be used to manage the IT infrastructure of an organization. This framework can be used to fulfill the requirements of COSO. The framework can be used by organizations to effectively manage their IT infrastructure, use it for achieving business objectives and to put in place the necessary controls to provide the information security. As security is a major concern, many international standards have been developed to provide necessary security controls to the information systems. Governments are now making it mandatory to incorporate the security controls so that the privacy of individuals and security of the governments are not compromised. To catch a thief, you have to think like a thief is an important lesson in security. So, hackers can do a better security testing. Testers called ‘ethical hackers’ who perform hacking without causing damage are now the most sought-after people for security testing.

Managing the Testing Process

251

Summary y The test manager has to plan the testing phase, execute the testing phase and carry out a postmortem analysis to improve the testing process.

y Test planning involves identification of test objectives, criteria for completion of testing, estimation of effort/time/budget, identification of team members and their training needs, identification of work products to be produced in testing phase, planning reviews and monitoring mechanisms and identification of metrics.

y Gantt (bar) chart helps in the overall project monitoring. The chart indicates the major activities and the time frames.

y Cost-Schedule-Milestone chart can be used to monitor the activities, timeframe and the effort/cost estimated and already spent.

y Dr. Meredith Belben’s team roles concept can be used effectively to form winning

teams. The team should have people who can play different roles which are actionoriented, people-oriented and thinking-oriented. Only when the team consists of individuals who excel in different roles, the team will achieve results.

y Margerison-McCann’s Team Management System is another model used for

formation of teams. In this model, every team member should be a team player and their work preferences should match with the team requirements. Eight types of personal profiles are defined in this model.

y Software metrics data has to be estimated during the planning stage and collected

during the test execution stage. Comparison of estimates with actual values helps in improving the testing process.

y Software defects have to be classified based on their severity. The severity levels

and the number of defects together are used to arrive at the criteria for completion of testing.

y MTBF, MTTR and MTTF are the parameters used to indicate the reliability of software.

y Configuration management involves finding out whether a proposed change will

have any adverse impact on the product or project. An impact analysis has to be done and a conscious decision has to be taken whether to accept the project or not.

y All information systems need to be tested thoroughly for security. Test engineers and managers need to be aware of various standards used for security testing of software systems. These standards include HIPAA, GLBA, COSO/ COBIT etc.

y Security testing carrying out a detailed vulnerability analysis of the information

assets. Based on this analysis, risk assessment has to be done and necessary security controls have to be put in place.

Chapter 7

252

Review Questions 1. ISO 9001 and CMMI are: a) Theoretical models

b) Models based on industry best practices

c) All models

d) Structure model

2. Quality Assurance manager has to report to: a) Project Manager

b) HR manager

c) CEO

d) COO

3. If a test engineer can test 4,000 lines of code per month, the effort required for a project of 200,000 lines of code is: a) 50

b) 40

c) Cannot be calculated

d) In negative

4. In Dr. Meredith Belber’s Team Role Concept, how many types of roles are defined? a) 3

b)

8

c) 6

d) 10

5. In Margerson-McCann Team Management System, how many profiles are defined? a) 3

b) 6

c) 8

d) 10

6. Gantt chart can be used to monitor the total effort already spent. a) True

b) False

7. Testing can be considered complete when: a) the total number of defects is above a threshold b) the total number of defects of a particular severity is below a threshold c) the total number of defects of a particular severity is above a threshold d) the total number of defects is equal to the threshold 8. Impact analysis means: a) Assessing the damage likely to be caused by the proposed change b) Assessing the impact on timeframe of the project only c) Assessing the impact on the budget of the project only d) None of the above 9. In SW-TMM level 1 organizations, testing and debugging are not differentiated. a) True

b) False

10. In SW-TMM, testing and debugging are differentiated at level: a) 1

b) 2

c) 3

d) 4

Managing the Testing Process

253 11. In SW-TMM, testing tools are used in organizations which are at level: a) 2

b) 3

c) 4

d) 5

12. In SW-TMM, quality parameters are tested by organizations which are at level: a) 2

b) 3

c) 4

d) 5

13. In SW-TMM, defects are tracked by organizations which are at level: a) 2

b) 3

c) 4

d) 5

14. In SW-TMM, effort and cost are planned and tracked by organizations which are at level: a) 2

b) 3

c) 4

d) 5

15. Hospital management system software has to be tested for: a) GLBA compliance

b) HIPAA compliance

c) DTDC

d) BIT

16. Penetration testing means: a) Testing whether the software is reliable b) Testing whether the software is usable c) Testing whether the software is vulnerable to attacks d) None of the above 17. Vulnerability analysis means: a) To check whether the information asset is prone to theft b) To check whether the information asset is prone to be damaged by cyclones c) To check whether the information asset is prone to cyber-attack d) All the above 18. It is unethical to employ ethical hackers for security testing: a) True

b) False

Index

388

Chapter

Software Testing Tools Chapter Objectives y y y y y y

8

Appreciate the need for automated testing tools Understand the requirements of testing tools Get to know the various categories of testing tools and their functionality Learn how to select a testing tool Learn the process of introducing the tools in an organization Learn how to use WinRunner, LoadRunner and TestDirector to get an idea of the power of testing tools

Software testing tools are of immense use to develop quality software. In this chapter, we will study the importance of these tools, various categories of testing tools and the process of introducing tools in your organization. How to use WinRunner, LoadRunner and TestDirector are also explained—to get an idea of the power of the testing tools and to illustrate how easy it is to carry out testing by using the tools.

8.1 Need for Tools Testing the software manually is time-consuming, error-prone and requires lot of infrastructure and manpower. All these drawbacks can be overcome if the testing process is automated. The testing tools reduce manual testing to a large extent and the testing can be done automatically. Using these tools has many advantages:

y Once the software is ready for testing, the functionality of the software can be tested repeatedly to improve the quality and the reliability.

y Testing can be done unattended, for example during nighttime and during holidays.

y When the software has to be tested in different environments (different hardware

platforms, different operating systems, using different browsers etc.), the labor involved can be reduced.

y Performance testing can be done without the need for many computers and many

test engineers. The test tools simulate the multiple users on a single machine. As compared to manual testing, finding out the transaction response times when multiple users access the same application will be very easy.

Chapter 8

256 y Testing process can be planned and managed effectively by using these tools. y Test reports can be generated automatically for later analysis and corrective action. y Testing can be done by using tools that are available to test not only generic

applications such as database applications and web sites, but also for DLLs, Visual Basic Programs, Siebel software, Power Builder software, databases, stand-alone Java applications etc.

y The testing process can be managed efficiently—the planning can be done systematically, the tests can be scheduled efficiently, and the bug tracking can be done effectively.

Hence, using automated test tools results in:

y y y y

Improvement in the quality and reliability of the software. Drastic reduction in time, effort and money spent on testing. A systematic approach to the testing process. Efficient management of the testing process even if the teams are at different geographical locations. Testing tools help in delivering high quality software and drastically reducing the testing time and effort.

Most of the commercial testing tools available are very sophisticated and you can learn how to use these tools in a matter of a few days.

8.2 Classification of Tools A wide variety of software testing tools are available to cater to the different types of software, different programming languages, and to carry out the different types of testing. These testing tools can be broadly divided into the following categories:

y Functional/Regression Testing Tools: These tools are used to test the application

software and web applications such as web sites. As majority of the applications involve Graphical User Interface (GUI), the tools test the GUI objects and functionality automatically. These tools carry out black box testing. Client/Server applications, Enterprise Resource Planning (ERP), software packages such as SAP, Customer Relations Management (CRM) software packages such as Siebel, web sites etc. can be tested for functionality by using these tools. Whenever a change is made to the software, the software needs to be retested and hence these tools are also called regression testing tools. Compuware’s QACenter, Segue Software’s

Software Testing Tools

257 SilkTest, IBM Rational’s Robot, Mecury Interactive’s WinRunner belong to this category.

y Performance Testing Tools: These tools are used to carry out performance testing

or stress testing. These tools are very useful to test how the application works when multiple users access the application simultaneously. The application can be for example, a database or a web site. These tools simulate multiple users on a single machine and hence you do not need many machines and many test engineers to do the performance testing. AutoTester’s AutoController, Compuware’s QALoad, Mercury Interactive’s LoadRunner, Segue Software’s SilkPerformer, IBM Rational’s Performance Tester, Apache JMeter belong to this category. Some other specialized tools in this category are Argogroup’s MonitorMaster for testing mobile applications that use WML and XHTML, Short Messaging Service (SMS) and Multimedia Messaging Service (MMS); IBM Rational’s Prevue-X and Prevue-ASCII for X-windows and ASCII terminal emulators.

y Source Code Testing Tools: These tools check the source code of the application

software. The testing is white box testing and hence the implementation details are taken into consideration. These tools are divided into two categories: 

Static analysis tools



Dynamic analysis tools

Static analysis tools are used to check the source code without executing it. These tools are used to find out a variety of problems in the source code such as:  

whether coding standards/guidelines are followed or not whether the code is portable or any machine/operating system dependent features have been used



variables declared but unused



dead code or unreachable code i.e., the code that will never be executed



obtaining the coding metrics such as the ratio of commented lines to uncommented lines, function point average etc.

Popular static analysis tools are IBM Rational Purify, McCabe Visual Quality ToolSet, Parasoft CodeWizard, QA C and CheckMate. The utility “lint” available in Unix/Linux systems is an excellent tool for checking the portability of C code as well as for analyzing the various problems listed above. Dynamic analysis tools find out the potential problems in the code, by executing the code. These tools are used (a) to detect memory errors such as memory allocation errors, memory leaks, buffer overflow problems (b) to check line coverage, branch/path/condition coverage (b) time taken for execution of different portions of the code which is required to carry out the code optimization etc. IBM Rational’s Purify is a popular tool for checking memory leaks. In Unix/Linux systems, “profilers” are available (lprof and prof commands)—these profilers display the number of times when each line is executed. The test engineer can

Chapter 8

258 study the profiler output to find out which portions of the code are not executed and then create test cases in such a way that the lines, which were not executed earlier, can be executed. Some application software source code testing tools are: AutomatedQA’s AQtime, Parasoft’s Insure++ and Telelogic’s Logiscope. Parasoft’s Insure++ is used to detect potential memory errors such as memory allocation, memory leaks. This tool inserts some test code (analysis functions) in the source code and then at run time, detects the potential errors in the code. This example brings out an interesting feature of some tools—additional code is inserted just for the purpose of automated testing.

y Java Testing Tools: As Java has become a popular programming language in recent years, a number of tools are available exclusively for testing Java applications. These tools are for testing applications written in Java programming language and for testing Java classes. Jemmy is an open source library to create automated tests for Java GUI applications. JMeter of Apache is another open source software to do performance testing. Parasoft’s jtest is used for Java class testing.

y Embedded Software Testing tools: Testing embedded software is a very

challenging task as the timing requirements for these applications are very stringent. In embedded systems, the code has to be optimized so that it occupies the minimum memory. IBM Rational Test Real Time is the widely used test tool in this category.

y Network Protocol Testing tools: As computer networks are becoming widespread,

testing networking/communication software has attained lot of importance in recent years. A number of tools are available for testing networking and communication protocols. Many test instrumentation vendors such as Agilent Technologies, Rhode & Schwartz etc. supply protocol analyzers which generate the necessary protocols based on international standards such as ITU-T standards. netIQ’s ANVL (Automated Network Validation Library) is to test routers and other networking products. This software generates packets in correct and incorrect formats to test the networking software. netIQ’s Chariot is a network performance testing tool.

y Configuration Management/Bug Tracking tools: In large software development

projects, configuration management is a very important process. Also, when the test engineers report bugs, the managers have to track these bugs and ensure that all the bugs are removed. To facilitate this activity, good workflow management software is important. Many such tools, which are web-based, are available. Bugzilla’s Bugzilla is an open source defect tracking system. Samba’s Jitterbug is a freeware defect tracking system. IBM Rational Software’s Clear DDTS is a change request management software. GNU’s GNATS is a freeware bug tracking and change management software. Segue Software’s SilkRadar is an issue tracking and bug tracking software. Microsoft Excel is also used extensively for bug tracking. In Unix/Linux/Solaris systems, utilities are available for version and release control

Software Testing Tools

259 of source code—these utilities are Source Code Control System (SCCS) and Revision Control System (RCS).

y Test process Management Tools: These tools help in managing process-oriented

software testing. Using these tools, the QA manager can create a formal test plan, allocate resources, schedule unattended testing, track the status of various bugs/activities etc. AutoTester’s AutoAdviser, Compuware’s QADirector, QIS’s QCIT, Segue Software’s SilkPlan Pro, IBM Rational Test Manager, Mercury Interactive’s TestDirector are some such tools.

y Support Tools: A number of tools are available that are not directly used by the test

engineers, but these tools help the developers/testers/QA engineers in different phases of software development. Some such tools are: 







Requirements management tools are used to manage the requirements. The tools help in keeping track of the priority of the requirement, whether each requirement is converted into design/code and whether the requirement is tested or not. Telelogic FOCAL POINT is a tool used extensively for requirements management. Design tools are used to carry out the analysis and the design of the software. A large number of tools are available for OOA and OOD, such as IBM Rational tools using which a detailed design can be carried out by using object-oriented analysis methodology. However, note that these tools are used by the developers/architects. Some of these tools are also capable of generating the test cases from the design. Telelogic TAU G2 is used to model the requirements of application software and Telelogic Rhapsody is used to model the requirements of embedded systems. Traceability tools help in tracing whether all the specifications have been tested by linking each specification with the corresponding test case. Unit testing tools are used to carry out the unit testing. Generally, only the developers carry out the unit testing, not the test engineers. The tools nunit (www.nunit.org) and junit (www.junit.org) are popular open source unit testing tools for C++ and Java programs respectively.

Read on to understand these tools in more detail. If your organization is planning to introduce the testing tools for the first time, you need to evaluate the commercially available tools for your specific application. Your choice for a specific tool is dependent on the application to be tested, hardware and software environment as well as the types of testing to be done.

8.2.1 Functional/Regression Testing Tools In functional testing of software, the functionality of the software is tested without bothering about the source code and implementation details. For example, consider the

Chapter 8

260 standard calculator software that is available on your Windows system. The GUI of the standard calculator is shown in Figure 8.1.

Figure 8.1 Standard Calculator GUI To test the functionality, you will do some calculations and find out whether you are getting the correct answers. You may type in ‘4 * 5 =’ and the correct answer 20 is displayed. You go on giving inputs and check the answers. If the expected result matches with the actual displayed result, your calculator software is working OK. While you are testing the software, you need to give test cases in such a way that all the buttons in the GUI are tested. So, you need to generate the test cases. Table 8.1 shows some test cases and the expected result. Note that we have chosen the test cases in such a way that each and every button on the GUI is pressed at least once. Some test cases are chosen in such a way that the calculations give an error output message (for example, finding the square root of a negative number). Test Case

Expected Output

4

0.25

1/x

-6

sqrt

Err: “Invalid input for function”

4

C

Clears the Display

1.2 * 3

3.6

5 / 2.0

2.5

7+8–9

6

600 * 2

%

12

2, MS, C, MR

2

MC, 2, M+, 3, M+, C, MR

5

Table 8.1 Test cases for testing the standard calculator

Software Testing Tools

261 While you are still in the development stage, your software is not likely to work perfectly, and for some test cases, you may get wrong results. During the course of development, you need to test the software many times by using the above test cases. If you do it manually, it takes lot of time, and yes, you will get bored. The functional/regression testing tools are of immense use in such situations. Using these tools, you can record your GUI operations. While running the testing tool, you invoke the application and give all the test cases as input. Automatically, the testing tool will record your inputs—keystrokes and mouse clicks—and then the tool will create a script. The script can be run subsequently and hence you are saved botheration of manually giving these inputs again and again. Consider another example of testing a login process. In many applications, initially a login screen will appear and you need to type in a username and a password. Only users who are authorized to use the application will be allowed to enter the application. Assume that a user is given the username ‘john’ and password as ‘mary’. To test this simple GUI, you need to generate the test cases. The specification says that the username and the password are case-sensitive. Hence, if the user types “John”, it is an invalid username. You can generate the test cases as shown in Table 8.2. Input: user name

Input: password

Expected output

john

mary

Login successful

John

Mary

Login unsuccessful

John

[no password typed]

Login unsuccessful

JOHN

MARY

Login unsuccessful

John

John

Login unsuccessful

Table 8.2 Test cases for Login screen As you can see, life is tough for a testing engineer. Just to test a login screen, you need to generate a good number of test cases. The development engineer, during the initial stages of development, would give his software with lot of bugs (the login will be successful if the password field is left blank). Every time, he modifies the software, you need to run all the above test cases and check whether the software still has any bugs. Obviously, if you record the test by using a tool, the test script can be run repeatedly. Now, testing the login screen is much more complicated. When a user enters his login name and password, the software will check in a database whether a valid username and password are entered. So, the login name can be not only john, but also donald, bush, prasad and so on. It is not possible to generate test cases by entering so many usernames and passwords manually. Testing tools provide a beautiful mechanism through data driven testing. In data driven testing, the username and password are considered as variables. The values these variables can take can be obtained from a database and the testing is carried out on them. Data driven testing is one of the most important features of

Chapter 8

262 these testing tools. As a test engineer, your job is just to create the test script with the variable names and then you can run the test in unattended mode (and go to a movie with your friend, and by the time you come back, your test report is ready). While testing the software in unattended mode, we get into another problem. Suppose, for an application, the user has to first login, then update a database table and then logout. You created the test cases for all the three functions. Suppose that the software fails at login stage itself—should you continue to run the test or stop the test. The testing tools provide you an option either to continue testing or stop testing. The application can be made to recover automatically to a specified state. This feature is called recovery management. All functional/regression testing tools operate in a similar fashion. They capture the user interactions with the GUI (keyboard entries and mouse clicks) and automatically create a test script. The script can be run repeatedly without the need for interacting with the GUI again and again. The scripting languages differ for each tool. To illustrate how to use a functional/regression testing tool, we will explain how to use WinRunner in this chapter. Some tool provide the facility of keyword-driven testing. The actions to be performed (called the action keywords) and the corresponding test data are stored in a database (say, in a spreadsheet). Tests can be defined using these keywords and the tests can be executed. The advantage of this approach is that the test engineers need not write test scripts. The functional testing tools that capture the user actions such as keyboard entries and mouse clicks, and replay these user actions by running the corresponding test script are called test robots or capture-replay tools. A wrong notion among the prospective testing professionals is that they do not get a chance to write any code. In fact, test engineers do write lot of test scripts and learning the scripting languages is very important. And, designing the tests is as creative as designing the software.

8.2.2 Performance/Load Testing Tools In Client/Server (C/S) applications, many users access the server application (such as a database) simultaneously. In such a case, the software has to be tested for its performance. Consider a database application. When only one user accesses the database, the transaction responses time to query the database and obtain the report is say, 10 seconds. When 10 users access the database simultaneously, what will be the response time—10 seconds or 40 seconds? When 1000 users access the database simultaneously what will be the response time—10 seconds or 10 minutes? No user will accept a few minutes of response time. Generally, in the SRS document, the response time will be specified. How do you test the software to find out the response time when say 100 users are simultaneously accessing the database? As shown in Figure 8.2, you need to connect 100 client systems and then put 100 test engineers on the job and use a communication

Software Testing Tools

263 network such as a Local Area Network (LAN) or a Wide Area Network (WAN). Your company will go bankrupt by the time you complete the testing. An economical approach is to use the performance testing tools which will do the job with minimal infrastructure and minimal number of persons.

Client#1

Communication Network Client#2

Client#100

Figure 8.2 Performance testing of a client/server application without using tools The performance testing tools functionality is shown in Figure 8.3. The server application continues to run on the server. But only one or a few clients will be running the client application but on these client machines, many virtual users are simulated, effectively stimulating an environment in which 100 users access the database.

Chapter 8

264 Virtual User 1 to 50

Client#1

Communication Network Server Virtual User 51 to 100

Client#2

Figure 8.3 Load Testing of client/server application using automated tools Today’s many enterprise applications such as ERP software, web services, web sites, CRM software etc. need to be tested for performance testing. Performance testing is also called load testing. These tools display the performance of the application by using graphs and charts so that analysis will be easy. For instance, you can plot a graph that shows the response time as a function of the number of simultaneous users who access the database. To illustrate how to use performance testing tools, we will study how to do performance testing by using Mercury Interactive’s LoadRunner. The performance testing tools create virtual users on the same machine and hence with minimal infrastructure you can simulate multi-user environment.

When timing analysis is done (for example, finding the response time in a database application) by using different tools, the timings may differ because different tools measure the time in different ways. In other words, the tool can have an impact on the outcome of the test. Such tools are called intrusive tools. The effect of these intrusive tools is called probe effect.

Software Testing Tools

265

8.2.3 Testing Process Management Tools Whether you are a manager or a test engineer, you need to follow a rigorous process for testing. This process involves working out a detailed test plan, working out the test cases, deciding the schedule for running various tests, generating the test reports, analyzing the test reports and tracking the bugs and checking whether the bugs have been removed and doing regression testing on the modified software. In addition, many managerial activities such as forming test teams, assigning responsibilities and target dates etc. need to be done to effectively manage the testing process. Many managers do all these activities on an ad-hoc basis; but making use of testing management tools will increase the productivity tremendously. Nowadays, many software companies have offices all over the world. Groups at different geographical locations work on the same project. The testing team and the development team may be located at different places. Even the testing groups may be located at different places. In such a case, the testing management tools can be used effectively for managing the testing process. Many commercially available testing management tools are web-based. A test engineer can log into the web site and update the defect report. The QA manager located at another place can login to the web site, check the status and assign the bug removal work to a developer. After the developer removes the bug, he can update the bug status to corrected. The web-based management tools provide the facility to create different users with different privileges to read or update the bugs database. Many tools provide the facility to generate emails and send the email to the person concerned. Testing management tools facilitate all these process-oriented activities to be done systematically. We will discuss how to use Mercury Interactive’s TestDirector in this chapter. For project managers, defect tracking is the most important activity. During the testing phase, a large number of defects are reported by the test engineers. Each defect has to be attended by the development engineer. Tracking these defects is a Herculean task and it must be done with the help of a tool. The defect tracking or bug tracking tools help in achieving this. The tools facilitate storing the information about the defect in a database and periodically updating the status. Each defect has to be put in a database and its details are recorded. These details include:

y y y y y y y

Who detected the defect? The entry criteria (status of the software just before the defect occurred). The exit criteria (status of the software after the defect). Date and time when the defect was found. Priority of the defect (how fast the defect has to be rectified). To whom the defect rectification been assigned. The status of the defect (rectification in progress, rectified, to be taken up later).

Chapter 8

266 y Target date for rectification. The database can be updated regularly by the engineers/managers. The manager can also get the necessary reports from the database and also generate the metrics which reflect the quality of the testing process. The bug tracking tools are also known as incident management tools. Testing process management tools can be integrated with other testing tools (functional/regression testing tools and performance testing tools). However, it is advisable to obtain the complete test suite from the same vendor rather than one tool from one vendor and another tool from another vendor.

8.3 Benefits of Tools The potential benefits of the tools are:

y In manual testing, a test engineer may test a particular functionality of software,

and report a bug. Sometimes, it is difficult for another test engineer to repeat the same test and check whether the same bug appears. In other words, the test may not be repeatable. If the testing is done by using a tool, the repeatability can be ensured as the test case and test results are stored.

y When testing is done manually, it may result in subjective evaluation of the

software. A test engineer may say vaguely the software is not reliable or the software is not working well. These subjective statements make the developers very angry. Using test tools, the testing can be made objective. For example, a test can be repeatedly done and then the reliability can be assessed.

y In manual testing, to carry out the regression testing, the same inputs have to be

repeatedly given and the output is checked. For human beings, this repetitive work is very boring and unproductive. Tools are extremely useful for regression testing.

y If Performance/load/stress testing is done manually for client/server and web

applications, lot of infrastructure (computers and network) and manpower are required. If tools are used, the testing can be done with minimal infrastructure and manpower.

y Managing the testing process manually is very difficult. Managing the process

involves estimating the resources, mapping the requirements to the test specifications, assigning priorities to the bugs, assigning the bugs to the test engineers for rectification, following up the status, obtaining the metrics for the testing process etc. All this can be done very efficiently by using the tools. Organizations in which there is no defined testing process will greatly benefit by using the tools.

y For software with thousands and even millions of lines of code, to check whether the source code is as per coding standards can be easily done through tools.

Software Testing Tools

267 y For white box testing and to ensure whether each and every line of code is tested,

tools (for example, the profilers) will be very handy, manual white box testing is next to impossible.

y The test documents (test plans, test logs, test results etc.) can be archived efficiently and managed by using configuration management tools.

y Test metrics can be easily measured, analyzed and archived. For example, the number of defects, the severity of defects, the number of incident reports, productivity of the test engineers, residual defects etc. can be easily collected and analyzed through statistical analysis tools and presented to the management through graphs and charts.

8.4 Risks Associated with the Tools Though using the tools has many benefits as discussed in the previous section, we need to remember that there are some risks too. In some organizations, after purchasing the tool, the expected benefits are not obtained and the management gets very angry and disappointed. We need to appreciate the following risks:

y The management does not estimate the time, effort and money required to

introduce a tool into the development process. The managers/QA specialists need to appreciate that in addition to the tool cost, the management has to spend on training the employees on the tool, carrying out a pilot project and if the pilot project is successful, to introduce the tool in the overall development process.

y When carrying out the pilot project, the success criteria need to be clearly defined.

If the criteria are not defined well, and the tool is introduced in the development process, there may be problems.

y Assuming that the tool will totally replace the smart test engineers is incorrect. For

some applications, manual testing may still be better. The tool may not be capable of doing all types of testing. Unrealistic expectations from the tool is one of the major risks.

8.5 Does your Organization Need Tools? Does your organization require an automated testing tool? This question can be very easily answered, if you spend some time and analyze the existing scenario in your organization:

y Are your customers happy with the software delivered by you? If they are unhappy and you get many reports on the bugs detected, the testing process needs to be improved a lot. Certainly, you need to consider the option of using a testing

Chapter 8

268 tool. The amount of effort and money you will be spending on corrective maintenance will be much more than the cost of buying the testing tools.

y If your customers are happy with the software you have delivered, find out the

effort, time and money spent on the testing phase. The productivity of the people can be increased many fold by using the tools. The test engineers can do a more productive work in generating the test cases, rather than unproductive data entry work by giving manual inputs.

y How many testing engineers are leaving your organization? If the manpower attrition rate is very high, then the most likely reason is that they were getting bored by doing repetitive things. Testing tools remove the manual work and the test engineers will enjoy the testing process.

y Using manual testing, you can never bring out a quality product if performance testing is a part of the requirements. If performance testing has to be done on your software, tools come in handy.

y If your organization believes in a process-oriented approach to testing, testing tools will be of immense use.

y If your project teams are located at different places, a web-based management tool will be very effective for an efficient management of the testing process.

Using metrics such as the cost of corrective maintenance, the effort/time/budget being spent on manual testing, and the manpower attrition in the testing division etc., you can estimate the effectiveness of the existing testing process. Testing tools certainly help in increasing the productivity of people and quality of the software.

8.6 Selecting Tools Once you decide to use testing tools, you need to make a choice based on the following criteria:

y What types of testing have to be done on the software? Invariably, most of the

application software packages need to be tested for functionality. As the software is likely to be modified, regression testing is also a must. Hence, you need to get functional/regression testing tools.

y If your software is a C/S application or a web application, you may need to do load testing. In such cases, you need to use a performance testing tool as well.

y Some vendors integrate testing management tools along with the above tools. Alternatively, some vendors supply a separate module at additional cost for testing process management.

y Once the types of testing tools required are decided, you need to obtain the tool for the environment in which the software runs. Your application may be running only

Software Testing Tools

269 in Windows, or only in Unix (or its variants such as Linux, Solaris etc.). Sometimes you may have to run the software in different platforms. In such cases, you need to check whether the testing tool supports such environments.

y Testing tool vendors also give tools that support testing for proprietary software packages such as Siebel, SAP etc.

y If you need to test the source code as well, you need to buy the source code testing tools.

y If you working on a very large software development project, tracking the bugs is a major issue. Bug tracking tools need to be used in such a case.

y If your organization is keen on implementing the process-oriented testing, testing

process management tools are a must. If the project teams are at different places, a web-enabled testing tool will be of great use.

In addition to the technical aspects, the manager needs to consider the following aspects:

y Whether the tool vendor has a local presence in your country or city . y Whether the vendor will provide enough support after purchase of the tools. The support should be in terms of providing the upgrades, rectifying any bugs in the software etc.

y Whether the vendor will provide training to your employees on the tools. y What is the cost of the tools? y What is the cost of yearly upgrades? Some vendors charge exorbitantly for yearly upgrades.

y What are the licensing terms? Is the license valid per site or per system? Is the license renewable yearly?

y Is there a provision for this tool for integration with tools of other vendors? A number of open source testing tools are now available. The main attraction of these tools is that they are free. However, you may not get technical support for such tools. The tools required for your organization need to be selected based on functionality required and testing environment (hardware and operating systems). You also need to take into consideration the various aspects such as the support and training provided by the vendor and the cost of the tools. A number of open source tools are available for free download from the Internet. They are free, but due to lack of support, they are yet to find wide acceptance.

Chapter 8

270

8.7 Introducing the Tools in the Testing Process In some organizations, when the management comes to know about the tools and some success stories, then the management will get excited and spend lot of money to buy the tools. Later on, the management will realize that the tools are not being effectively used and all the money is a waste. It is very important for the management to follow a systematic procedure for introducing a tool in the organization, as shown in Figure 8.4. Assess existing testing process, strengths and weaknesses

Identify the type of tools required

Evaluate commercially available / open source tools and select the required tool

Do a pilot project using the tools

Yes

Is the pilot project successful?

Introduce the tools across the organization

No

Analyze the reasons for failure

No

Is there is a need for another pilot project?

Stop

Figure 8.4 Process of introducing a tool in the organization

Yes

Software Testing Tools

271 y You need to assess the present testing process and find out the strengths and

weaknesses. Metrics such as how much effort/money is being spent on the product after it is delivered to the customer, how many bugs were reported by the customer and the severity of these bugs, how many test engineers are leaving the organization etc., will reflect how good or how bad your testing process is.

y Depending on the project requirements, identify the types of tools required for your organization.

y Evaluate the different commercially available tools as well as open source tools and select the required tools.

y Using the selected tools, carry out a pilot project. Identify the criteria for evaluating the effectiveness of the tools.

y If the pilot project is successful, introduce the tools for other projects. If the pilot project is not successful, then analyze the need for another pilot project.

In the following sections, we will study how to use WinRunner, LoadRunner and TestDirector. Knowledge of vendor-specific tools is not a must for obtaining certifications in software testing from ISTQB/ISEB/QAI. The following sections are given for a good understanding of using the tools. Even if, you do not have access to the tools, going through the following sections will give a good exposure to the use of tools and also the testing process.

8.8 WinRunner Mercury Interactive’s WinRunner is a testing tool to do functional/regression testing. Using WinRunner, you can record GUI operations. While recording, WinRunner automatically creates a test script. This test script can be run automatically later on for carrying out the unattended testing. The important aspects of WinRunner are as follows:

y You can do functional/regression testing of a variety of application software

written in programming languages, such as PowerBuilder, Visual Basic, C/C++ and Java. You can also carry out the testing on ERP/CRM software packages.

y You can do the testing in all flavors of Windows operating systems and different browser environments such as Internet Explorer and Netscape Navigator.

y You can record the GUI operations in the record mode. WinRunner automatically creates a test script. This test can be modified if required and can be executed later on in unattended mode. The recovery manager enables the application to be brought to a known state in case there is a problem during the unattended testing. Rapid Test Script Wizard creates the test scripts automatically.

y You can add checkpoints to compare actual and expected results. The checkpoints can be GUI checkpoints, bitmap checkpoints and web links.

Chapter 8

272 y It provides a facility for synchronization of test cases. y Data Driver Wizard provides the facility to convert a recorded test into a data driven test. So, you can replace data with variables within a test script. For example, you can test a login process by taking the input for username and password fields from a database.

y Database checkpoints are used to verify the data in a database during automated testing. The records that are inserted, deleted, modified or updated will be highlighted so that you can ensure database integrity and transaction accuracy.

y The Virtual Object Wizard of WinRunner is used to teach WinRunner to recognize, record and replay custom objects.

y The reporting tools provide the facility to generate automatically the test reports and analyze the defects.

y WinRunner can be integrated with the testing management tool TestDirector to automate many of the activities in the testing process.

8.8.1 Testing an Application by using WinRunner After installing the WinRunner on your computer, invoke the WinRunner application in the following way: Start|Programs|WinRunner|WinRunner The opening screen of the WinRunner application is displayed, prompting you to select one of the three options, as shown in Figure 8.5:

y New Test: To create a new test script; y Open Test: To open an existing test script; y Quick Preview: To view the quick preview of WinRunner.

Figure 8.5 Startup Screen of WinRunner

Software Testing Tools

273 You can select the Quick Preview to get a good understanding of this tool. You can run the sample application given along with this tool to understand the intricacies of this tool.

Recording Test Cases To test any application, first you can run the application and understand its operation. Then, you can invoke WinRunner, again run the application and record the GUI operations. During the recording mode, WinRunner will capture all your actions, which button you pressed, where you clicked the mouse, etc. You need to work with the application as usual and perform all the actions to be tested. Once the recording is completed, WinRunner generates a script in TSL (Test Script Language). You can run this test script generated by WinRunner to view the results. The test results will show whether the test has passed or failed. There are two modes of recording:

y Context Sensitive mode: This mode of recording is used when the location of the GUI controls (i.e. X and Y coordinates) or the mouse positions are not necessary.

y Analog mode: This mode of recording is used when the mouse positions, the

location of the controls in the application, also play an important role in testing the application. This mode of recording has to be used to validate bitmaps, testing the signature etc.

Recording a Test Case The procedure for recording a test case is as follows: 1. Open a new document: File|New (or select New Test from the WinRunner’s Welcome screen). 2. Open (run) the application to be tested. 3. Start recording a test case: Create|Record|Context Sensitive (or click on the toolbar’s Record button ( ) once) to record in Context Sensitive mode. 4. Select the application to be tested by clicking on the application’s title bar. 5. Perform all the actions to be recorded. 6. Once all required actions are recorded, stop the recording: Create|Stop (or click on the toolbar’s Stop button ( ) once) to stop the recording. WinRunner generates the script for the recoded actions. There are two modes for generating the test cases|Global GUI map file mode and GUI map file per test mode. By default, it is in Global GUI map file mode.

y In Global GUI map file mode, you have to explicitly save the information learnt by WinRunner. WinRunner saves it in a file with extension .gui.

Chapter 8

274 When you have to run a test, you need to load the corresponding GUI map file; otherwise it will not be able to recognize the objects in the test case and displays an error message.

y In GUI map file per test mode, WinRunner automatically saves the information it has learnt from the application.

It is always preferred to work in Global GUI map file mode. Let us understand the various procedures.

Saving the GUI Map File The procedure for saving the GUI map file in Global GUI map file mode is as follows: 1. Record a test case by following the procedures given in the section ‘Procedure for Recording a Test Case’. 2. Open the GUI Map Editor window (Figure 8.6): Tools|GUI Map Editor

Figure 8.6 Selecting GUI Map Editor 3. On selecting the GUI Map Editor, the GUI Map Editor dialog box, as shown in Figure 8.7, gets displayed.

Software Testing Tools

275

Figure 8.7 GUI Map Editor dialog 4.

Save the GUI Map file: File|Save As

5. When a File dialog box appears, enter the filename. 6. Close the GUI Map Editor window.

Loading the GUI Map File The procedure for loading the GUI map file is as follows: 1. Open the GUI Map Editor: Tools|GUI Map Editor 2. Close all the opened GUI Map files: File|Close All 3. Select the required GUI Map file: File|Open A File Open dialog box appears from which you have to select the GUI map file to be loaded. Once the file is loaded, it displays the list of items it has learnt during the recording phase, as shown in Figure 8.8.

Chapter 8

276

Figure 8.8 GUI Map Editor with list of items learnt

Running a Test Case The procedure for running a test case is as follows: 1. Open the test script to be executed. 2. Run the test: Run|Run from top (or press F5). WinRunner executes the generated script and displays the results in the Test Results window.

8.8.2 Testing the Standard Calculator Application We will now test the Standard Calculator application available on your Windows system using WinRunner. You can invoke the calculator application from the desktop this way: Start|Programs|Accessories|Calculator. The GUI of the Calculator application gets displayed, as shown in Figure 8.1. To test the complete functionality of the application, we need to generate test cases in such a way that all the buttons are made use of. We need to generate some test cases which will give correct output and also some test cases which will give error messages. Table 8.3 presents such test cases and the expected output for each test case. Test Case

Expected Output

4

0.25

-6

1/x sqrt

Err: “Invalid input for function”

Software Testing Tools

277 Test Case

Expected Output

4

Clears the Display

C

1.2 * 3

3.6

5 / 2.0

2.5

7+8–9

6

600 * 2

%

12

2, MS, C, MR

2

MC, 2, M+, 3, M+, C, MR

5

Table 8.3 Test cases and expected outputs for testing calculator

Testing the Functionality of Application Let us now understand the test cases to test the functionality of the application.

Test Case #1: Inverse Operation To test the Inverse operation (inverse of 4 using 1/x button), perform the following steps: 1. Open WinRunner application. 2. Open Calculator application. 3. Create a new document (Figure 8.9): File|New (either click New button ( toolbar or press Ctrl+N).

Figure 8.9 Creating a new document in WinRunner

) on

Chapter 8

278 4. To start recording: Create|Record-Context Sensitive (you can also press F2 or click Record-Context Sensitive button ( ) on the toolbar, as shown in Figure 8.10). Figure 8.11 shows selection of ‘Record-Context Sensitive’ option from the ‘Create’ menu.

Figure 8.10 Recording the test case using the Record button

Figure 8.11 Recording the test case using the menu item 5. Select the Calculator application to start recording these actions: 

Click 4 on the Calculator application



Click the 1/x button on the Calculator application to find the inverse of 4.



The result, 0.25, will be displayed on the Calculator application.

6. To stop the recording process: Create|Stop Recording (or click Stop button ( ) on the toolbar of WinRunner, as shown in Figure 8.12). Figure 8.13 shows the selection of the ‘Stop Recording’ option from the ‘Create’ menu.

Figure 8.12 Stop Recording button

Software Testing Tools

279

Figure 8.13 Stop Recording menu item 7. Once the recording is over, WinRunner generates TSL Script, as shown in Figure 8.14.

Figure 8.14 TSL test script 8. Save the file as ‘inverse’ in the selected folder: File|Save. In the Save dialog box that appears, save the test script with name ‘inverse’. 9. Run the test script generated by WinRunner: Run|Run from Top (either press F5 or click Run from Top button ( ) on the toolbar of WinRunner as shown in Figure 8.15). Figure 8.16 shows the ‘Run from Top’ option selected from the ‘Run’ menu.

Chapter 8

280

Figure 8.15 Running the Test script using Run from Top button

Figure 8.16 Running the Test script using Run from Top menu item 10. After executing the TSL statements, WinRunner generates test results, as shown in Figure 8.17.

Figure 8.17 Test results for Test Case #1 The Results column indicates whether the test has Passed or Failed. The test results also give useful information, such as the line numbers in the test script, name of the test case and details, the status of result, and the time taken for executing the test case.

Test Case #2: Square Root Operation Let us now test the operation Square root of -6. You can use the same procedure explained in section on Test Case #1 for recording the test case.

Software Testing Tools

281 The following Test script will be generated: # Calculator win_activate ("Calculator"); set_window ("Calculator", 1); obj_mouse_click ("Button_38", 20, 12, LEFT); obj_mouse_drag ("Button_35", 10, 15, 11, 14, LEFT); obj_mouse_click ("Button_60", 20, 11, LEFT);

When you run the Test script again, the test results will be displayed, as shown in Figure 8.18.

Figure 8.18 Test results for Test Case #2

Test Case #3: Clearing Display Let us now clear the display after performing some operations. For this test case, the following script will be generated: # Calculator win_activate ("Calculator"); set_window ("Calculator", 3); obj_mouse_click ("Button_37", 13, 23, LEFT); obj_mouse_click ("Button_59", 34, 19, LEFT);

Again, run the test script and observe the test results (Figure 8.19).

Figure 8.19 Test results for Test Case #3

Chapter 8

282 Test Case #4: Multiplication Operation Let us now perform multiplication of two numbers: (1.2 * 3) When you record this test case, the following test script is generated: # Calculator win_activate ("Calculator"); win_move ("Calculator", 489, 176); set_window ("Calculator", 5); obj_mouse_click ("Button_29", 11, 15, LEFT); obj_mouse_click ("Button_40", 16, 13, LEFT); obj_mouse_click ("Button_34", 26, 12, LEFT); obj_mouse_click ("Button_43", 10, 4, LEFT); obj_mouse_click ("Button_39", 20, 9, LEFT); obj_mouse_click ("Button_50", 23, 19, LEFT);

The test results are shown in Figure 8.20.

Figure 8.20 Test results for Test Case #4 You can follow the same procedure for other test cases.

Test Case #5: Division Operation Similarly, you can perform division of two numbers: (5/2.0)

Test Case #6: Operation (7+8-9) Similarly, test the operation: (7 + 8 – 9)

Test Case #7: Percentage Operation Similarly, test the operation: 2% of 600

Software Testing Tools

283 Test Case #8: Buttons Similarly, test the MS and MR buttons.

Test Case #9: Button Similarly, test M+ and MR buttons.

Test Case #10: Button Similarly, test MC button

Test Case #11: Backspace Similarly, test the Backspace. If you want to repeatedly execute the above test cases, it will be very time consuming. Instead, you can call the test cases by using the call function as described in the next section.

Calling the Test Cases using call Function The call function can be used to execute a series of test cases without any user interaction. The syntax of call function is as follows: call