Testing Approaches

 

In a large software project it is impossible to integrate all the modules in one shot and start testing the software as a whole. The system has to be built in stages and the product has to built incrementally carrying out testing on each incremented software. The testing approaches discussed in this article help in achieving the objective.

Static Testing Vs. Dynamic Testing:

Once the code is written, there are two ways of testing - Static Testing & Dynamic Testing.

In static testing the code is checked without executing it through Reviews/Inspections/Walk-through, hence it is also known as desk checking. This can be done either formally in a meeting or informally by two or more development/QA engineers. This testing is mainly performed to test the documents related to software, QA/QC team go through the source code/document or use certain debugging tools to find out the syntax errors. In dynamic testing the code is executed and the actual results are compared with the expected results. It is always suggested to carry out both the types of testing because each testing reveals specific bugs. For instance, if the development team forgot to implement a particular user requirement, this can be only revealed in static testing as the code is checked for completeness. But dynamic testing does not reveal such errors.

Positive Testing Vs. Negative Testing:

The software has to do what it is intended to do and nothing more or less. We need to test the software to check whether it is doing what it is intended to do. But that is not enough. We also need to check whether it is doing anything, which is not intended to do. This is differentiated  by Positive testing & Negative testing. 

Positive testing is testing the software to ensure that it does what it is intended to do. Negative testing is testing the software to ensure that it does not do anything, it is not intended to do.

Top-down Testing Vs. Bottom-up Testing:

In top-down approach testing is done from the top of hierarchy. Dummy routines called stubs are introduced that simulate a module.

In bottom-up approach, testing is done from the bottom of hierarchy. Dummy routines called drivers are introduced that invoke a module.

At system level, carrying out top-down testing is difficult, it is better to do the testing using bottom-up approach - test the sub-modules, the sub-systems and the total system.

Generally, the top-down approach is followed for design and bottom-up approach for testing.

Functional Testing Vs. Structural Testing:

In functional testing we test what the software does. Here the functionality of the module is tested and structure is not considered. Test cases are based on specifications and internals of modules are  also not considered. In functional testing we try to find discrepancies between how the program performs and its specification. The test data are derived exclusively from the specification because we do not have access to the program code and you do not have knowledge of the internal structure of the program. This type of testing is also known as "Black-Box testing".

Structural is a "White Box" testing based on the algorithm or code. The structure of a program differs at different levels. At component level, the structure is the code. At integration level, the structure is the tree diagram that shows the interconnecting modules. At system level, the structure is the menu structure or web page structure or business process depending on the application.

Structural testing is also called clear-box testing and glass box testing. 

Structural testing at component level is used to test the implementation of the program. The source code is looked into for testing the software. Structural testing involves:

1. statement coverage/decision coverage
2. condition coverage/branch coverage
3. path coverage/function coverage

Re-testing Vs. Regression Testing:

Re-testing is done to make sure that bug is fixed and failed functionality is working fine or not, this is kind of verification method followed in testing field for the fixed bugs. Re-testing is used only for failed test cases. Re-testing cannot be automated. It is also known as planned testing.

Regression testing is a type of software testing, which is used to verify that modifications in the software or the environment have not caused any unintended adverse side effect. Passed test cases can be executed and this can be automated or done manually. Regression testing is known as a generic testing.

Few Important Testing Definations

• What is a Bug?

When the actual result deviates from the expected result, while testing a software application or product then it results into a defect. Hence, any deviation from the specification mentioned in the product functional specification document is a defect. In different organisations it is also called as a Bug. 

• What is a Defect? 

If software misses some feature or function, from what is there in requirement it is called as a defect. 

• What is CMM? 

The Capability Maturity Model for Software (CMM or SW-CMM) is a model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of these processes. 

• What is Beta Testing? 

Beta testing is testing of a release of a software product conducted by customers. 

• What is Black Box Testing? 

Testing based on an analysis of the specification of a piece of software without reference to its internal workings. The goal is to test how well the component conforms to the published requirements for the component. 

• What is Bottom Up Testing?

An approach to integration testing where the lowest level components are tested first, then used to facilitate the testing of higher level components. The process is repeated until the component at the top of the hierarchy is tested.

• What is Boundary Testing?

Test which focus on the boundary or limit conditions of the software being tested. (Some of these tests are stress tests).

 

• What is Boundary Value Analysis?

BVA is similar to Equivalence Partitioning but focuses on "corner cases" or values that are usually out of range as defined by the specification. This means that if a function expects all values in range of negative 100 to positive 1000, test inputs would include negative 101 and positive 1001.

 

• What is Branch Testing?

Testing of all branches in the program source code is called Branch Testing.

 

• What is Coding?

The generation of source code is called Coding. 

 

• What is Compatibility Testing?

Testing whether software is compatible with other elements of a system with which it should operate, e.g. Browsers, Operating Systems, Hardware.

 

• What is a Component?

A component is an identifiable part of a larger program or construction. Usually, a component provides a particular function or group of related functions.

 

• What is Component Testing?

Testing of individual software components is called Component testing.

 

• What is Acceptance Testing?

Testing conducted to enable a user/customer to determine whether to accept a software product. Normally performed to validate the software meets a set of agreed acceptance criteria.

• What is Accessibility Testing?

Verifying a product is accessible to the people having disabilities (deaf, blind, mentally disabled etc.). 

 

• What is Ad-hoc Testing?

A testing phase where the tester tries to 'break' the system by randomly trying the system's functionality is called Ad-hoc testing. This can include negative testing also.

• What is Agile Testing?

Testing practice for projects using agile methodologies, treating development as the customer of testing and emphasizing a test-first design paradigm. See also Test Driven Development.

• What is Application Binary Interface (ABI)?

A specification defining requirements for portability of applications in binary forms across different system platforms and environments is called Application Binary Interface (ABI).

• What is Application Programming Interface (API)?

A formalized set of software calls and routines that can be referenced by an application program in order to access supporting system or network services is called Application Programming Interface (API).

• What is Automated Software Quality (ASQ)?

The use of software tools, such as automated testing tools, to improve software quality is called Automated Software Quality (ASQ).

• What is Automated Testing?

Testing employing software tools which execute tests without manual intervention is called Automated Testing. Can be applied in GUI, performance, API, etc. testing. The use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions.

• What is Backus-Naur Form?

It is a meta-language used to formally describe the syntax of a language.

• What is Basic Block?

A sequence of one or more consecutive, executable statements containing no branches is called Basic Block.

• What is Basis Path Testing?

A white box test case design technique that uses the algorithmic flow of the program to design tests.

• What is Basis Set?

The set of tests derived using basis path testing. 

• What is Baseline?

The point at which some deliverable produced during the software engineering process is put under formal change control.

• What is Binary Portability Testing?

Testing an executable application for portability across system platforms and environments, usually for conformation to an ABI specification is called Binary Portability Testing.

• What is Breadth Testing?

A test suite that exercises the full functionality of a product but does not test features in detail is called Breadth Testing.

• What is CAST?

Computer Aided Software Testing refers to the computing-based processes, techniques and tools for testing software applications or programs.

• What is Capture/Replay Tool?

A test tool that records test input as it is sent to the software under test. The input cases stored can then be used to reproduce the test at a later time. Most commonly applied to GUI test tools.

• What is Cause Effect Graph?

A graphical representation of inputs and the associated outputs effects which can be used to design test cases.

• What is Code Complete?

Phase of development where functionality is implemented entirety; bug fixes are all that are left. All functions found in the Functional Specifications have been implemented.

• What is Code Coverage?

An analysis method that determines which parts of the software have been executed (covered) by the test case suite and which parts have not been executed and therefore may require additional attention.

• What is Code Inspection?

A formal testing technique where the programmer reviews source code with a group who ask questions analysing the program logic, analysing the code with respect to a checklist of historically common programming errors and analysing its compliance with coding standards. Know more about the Inspection in software testing.

• What is Code Walkthrough?

A formal testing technique where source code is traced by a group with a small set of test cases, while the state of program variables is manually monitored, to analyse the programmer's logic and assumptions. Know more about Walkthrough in software testing.

• What is Concurrency Testing?

Multi-user testing geared towards determining the effects of accessing the same application code, module or database records. Identifies and measures the level of locking, deadlocking and use of single-threaded code and locking semaphores.

• What is Conformance Testing?

Conformance testing, also known as compliance testing, is a methodology used in engineering to ensure that a product, process, computer program or system meets a defined set of standards.

• What is Context Driven Testing?

The context-driven school of software testing is flavour of Agile Testing that advocates continuous and creative evaluation of testing opportunities in light of the potential information revealed and the value of that information to the organization right now.

• What is Conversion Testing?

Testing of programs or procedures used to convert data from existing systems for use in replacement systems. 

• What is Cyclomatic Complexity?

A measure of the logical complexity of an algorithm, used in white-box testing is called Cyclomatic Complexity.

• What is Data Dictionary?

A database that contains definitions of all data items defined during analysis.

• What is Data Flow Diagram? 

A data flow diagram (DFD) is a graphical representation of the "flow" of data through an information system, modelling its process aspects.

• What is Data Driven Testing?

Testing in which the action of a test case is parameterised by externally defined data values, maintained as a file or spreadsheet. This is a common technique used in Automated Testing.

• What is Debugging?

The process of finding and removing the causes of software failures is called Debugging.

• What is Defect?

When actual result deviates from the expected result while testing a software application or product then it results into a defect.

• What is Dependency Testing?

Dependency Testing, a testing technique in which an application's requirements are pre-examined for an existing software, initial states in order to test the proper functionality. The impacted areas of the application are also tested when testing the new features or existing features.

• What is Depth Testing?

A test that exercises a feature of a product in full detail is called Depth testing

• What is Dynamic Testing?

Testing software through executing it is called Dynamic testing. Also know about Static Testing.

• What is Emulator?

A device, computer program, or system that accepts the same inputs and produces the same outputs in a given system is called Emulator.

• What is Endurance Testing?

Checks for memory leaks or other problems that may occur with prolonged execution.

• What is End-to-End testing?

Testing a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate is called End-to-End testing.

• What is Equivalence Class?

A portion of a component's input or output domains for which the component's behaviour is assumed to be the same from the component's specification.

• What is Equivalence Partitioning?

A test case design technique for a component in which test cases are designed to execute representatives from equivalence classes.

• What is Exhaustive Testing?

Testing which covers all combinations of input values and preconditions for an element of the software under test.

• What is Functional Decomposition?

A technique used during planning, analysis and design; creates a functional hierarchy for the software.

• What is Functional Specification?

A document that describes in detail the characteristics of the product with regard to its intended features is called Functional Specification.

• What is Functional Testing?

Testing the features and operational behaviour of a product to ensure they correspond to its specifications. Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions or Black Box Testing.

• What is Glass Box Testing?

White Box Testing is also known as Glass Box testing.

• What is Gorilla Testing?

Gorilla Testing is a testing technique in which sometimes developers also join hands with testers to test a particular module thoroughly in all aspects.

• What is Gray Box Testing?

A combination of Black Box and White Box testing methodologies is called Gray box testing. Testing a piece of software against its specification but using some knowledge of its internal workings.

• What is High Order Tests?

Black-box tests conducted once the software has been integrated.

• What is Independent Test Group (ITG)?

A group of people whose primary responsibility is software testing. Know more about Independent testing.

• What is Inspection?

A group review quality improvement process for written material. It consists of two aspects; product (document itself) improvement and process improvement (of both document production and inspection).

• What is Integration Testing?

After integrating two different components together we do the integration testing. Integration testing is usually performed after unit and functional testing. This type of testing is especially relevant to client/server and distributed systems.

• What is Installation Testing?

Installation testing is a kind of quality assurance work in the software industry that focuses on what customers will need to do to install and set up the new software successfully. The testing process may involve full, partial or upgrades install/uninstall processes.

• What is Load Testing?

A load testing is a type of software testing which is conducted to understand the behaviour of the application under a specific expected load. Also see Performance Testing.

• What is Localization Testing?

This term refers to making software specifically designed for a specific locality.

• What is Loop Testing?

A white box testing technique that exercises program loops in order to validate them.

• What is Metric?

Metric is a standard of measurement. Software metrics are the statistics describing the structure or content of a program. A metric should be a real objective measurement of something such as number of bugs per lines of code. 

• What is Monkey Testing?

Testing a system or an Application on the fly, i.e just few tests here and there to ensure the system or an application does not crash out.

• What is Negative Testing?

Testing aimed at showing software does not work. This is also known as "test to fail". See also Positive Testing.

• What is Path Testing?

Testing in which all paths are in the program source code are tested at least once.

• What is Performance Testing?

Testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users.

• What is Positive Testing?

Testing aimed at showing software works. This is also known as "test to pass".

• What is Quality Assurance?

All those planned or systematic actions necessary to provide adequate confidence that a product or service is of the type and quality needed and expected by the customer.

• What is Quality Audit?

A systematic and independent examination to determine whether quality activities and related results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives.

• What is Quality Circle?

A group of individuals with related interests that meet at regular intervals to consider problems or other matters related to the quality of outputs of a process and to the correction of problems or to the improvement of quality.

• What is Quality Control?

The operational techniques and the activities used to fulfil and verify requirements of quality.

• What is Quality Management?

That aspect of the overall management function that determines and implements the quality policy.

• What is Quality Policy?

In quality management system, a quality policy is a document jointly developed by management and quality experts to express the quality objectives of the organization, the acceptable level of quality and the duties of specific departments to ensure quality.

• What is Quality System?

The organizational structure, responsibilities, procedures, processes, and resources for implementing quality management is called Quality System.

• What is Race Condition?

A race condition is an undesirable situation that occurs when a device or system attempts to perform two or more operations at the same time, but because of the nature of the device or system, the operations must be done in the proper sequence to be done correctly.

• What is Ramp Testing?

Continuously raising an input signal until the system breaks down.

• What is Recovery Testing?

Recovery testing confirms that the program recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.

• What is Regression Testing?

Retesting a previously tested program following modification to ensure that faults have not been introduced or uncovered as a result of the changes made is called Regression testing.

• What is Release Candidate?

A pre-release version, which contains the desired functionality of the final version, but which needs to be tested for bugs (which ideally should be removed before the final version is released).

• What is Sanity Testing?

Brief test of major functional elements of a piece of software to determine if its basically operational. See also Smoke Testing.

• What is Scalability Testing?

Performance testing focused on ensuring the application under test gracefully handles increases in work load.

• What is Security Testing?

Testing which confirms that the program can restrict access to authorised personnel and that the authorised personnel can access the functions available to their security level.

• What is Smoke Testing?

In Software testing context, smoke testing refers to testing the basic functionality of the build.

• What is Soak Testing?

Running a system at high load for a prolonged period of time is called Soak testing. For example; run several times more transactions in an entire day (or night) than would be expected in a busy day, to identify the performance problems that appear after a large number of transactions have been executed.

• What is Software Requirements Specification?

A deliverable that describes all data, functional and behavioural requirements, all constraints, and all validation requirements for software.

• What is Software Testing?

A set of activities conducted with the intent of finding errors in software.

• What is Static Analysis?

Analysis of a program carried out without executing the program.

• What is Static Analyser?

A tool that carries out static analysis is called Static Analyser.

• What is Static Testing?

Analysis of a program carried out without executing the program.

• What is Storage Testing?

Testing that verifies the program under test stores data files in the correct directories and that it reserves sufficient space to prevent unexpected termination resulting from lack of space. This is external storage as opposed to internal storage.

• What is Stress Testing?

Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine the load under which it fails and how. Often this is performance testing using a very high level of simulated load.

• What is Structural Testing?
  

Testing based on an analysis of internal workings and structure of a piece of software. See also White Box Testing. 

 

Software Testing Levels

Levels of Testing

To get a quality software product, developers and testers need to test the software at different levels during the coding/testing phase. 

During the design stage the software product is divided into components or units and each components/unit is developed. Similarly, while testing it is not good if we combine all the units into modules and all the modules into the system in single shot and then start testing the entire system. A practical approach is to divide the testing process into different levels. Each components/unit has to be tested separately, the modules have to be built from the units, which are then tested. The modules are then combined together and the system is built and tested. Following are the different testing levels: 

• Unit Testing

• Integration Testing

• System Testing

• Acceptance Testing

As shown in below figure, each level of testing is for testing a specific entity. Unit testing is done to test the source code. Integration testing is done to test the design. System testing is done to test the system against SRS and finally Acceptance testing is done to test the software against client/user requirements. To check whether the software meets the operational/business needs of the client, the software product is installed in the actual working environment and tested.

Figure: Levels of testing

Unit Testing/Component Testing

Unit/component is the smallest entity in the software product. Every unit has to be tested separately to check whether it is as per the specifications. It is a software design and development method, where the programmer gains confidence that individual unit of source code are fit for use. Unit/Component testing is conducted by developers to verify the software at unit level. Unit tests typically exercises particular modules of software code. Developers utilise debuggers or self-developed tools to verify functionality. Unit/component testing occur before the software reaches to software testers.

Testing of a collection of units/components that have a logical relationship is called string testing. Suppose you have developed the code for a few data structures (a stack, a queue, and a tree), you may like to test all these units one after the other - this is string testing.

It may not be possible to test a unit/component in isolation. Developers may need to write additional code (stubs, drivers, simulators) to do the testing.

Module Testing

A module is an independent entity in software product. The tested units are integrated into a module and each module is tested separately for the specification.

Module testing is largely a white box oriented. The objective of doing Module, testing is not to demonstrate proper functioning of the module but to demonstrate the presence of an error in the module.

Module level testing allows to implement parallelism into the testing process by giving the opportunity to test multiple modules simultaneously. Testing multiple module/sub-systems in parallel is also known as parallel testing. 

Integration Testing

After the modules are tested, the modules can be integrated together. During integration testing a very systematic approach has to be followed. Without following any systematic approach if you integrate all the modules together in one shot and then start testing then it is called big bang testing. 

 

Foremost the first thing is to identify which modules can be integrated first. The interfaces between the modules should be well defined to achieve this. 

The purpose of this level of testing is to expose defects in the interaction between these software modules when they are integrated.  Integration Testing focuses on checking data communication amongst these modules. Hence it is also termed as 'I & T' (Integration and Testing), 'String Testing' and sometimes 'Thread Testing'.

System Testing

Once all the modules are integrated the system is ready for a full-fledged testing. It allows checking system's compliance as per the requirements. It tests the overall interaction of components. It involves load, performance, reliability and security testing.

System testing most often the final test to verify that the system meets the specification. It evaluates both functional and non-functional requirements for the testing. The application is tested in an environment that is very close to the production environment where the application will be deployed. This type of testing is performed by a specialized testing team.

Acceptance Testing

After the system testing is completed in your premises the software has to be approved by the client. The customer will carry out a rigorous testing called Acceptance testing. It is conducted to find if the requirements of the specification or contract are met as per its delivery. Basically done by the user or customer and other stockholders can be involved in this process.

Some Other Types of Testings

• Regression Testing

• Buddy Testing

• Alpha Testing

• Beta Testing