Software Testing Practices in IT Industry of Pakistan

Abstract

A software must do what it intends to do. The quality is core factor in IT Industry and in software products. The quality of the product is main concern of the producer and the main requirement of a customer. Software testing is the core activity of quality assurance and very important phase of software development life cycle. Quality of any system is known by testing that product. With the advancement of IT industry many new verification and validation technique are introduced over a time. New trends are well adopted in developed countries like USA, UK etc. But not as well in IT of developing countries like Pakistan. The need of this paper is due to a large number of outsourced projects and also multinational IT companies in Pakistan. Due to high demand of outsourced projects the software testing become an integral part of IT Industry of Pakistan. Our team has collected data about testing in IT Industry of Pakistan which will be benefit for the testers in IT Industry not just in Pakistan but worldwide. These results could be use by researchers to identify the strengths and weakness of testing in IT industry.

Full text

Software Testing Practices in IT Industry of Pakistan
Muhammad Shah Jahan
Department of Computer & Software Engineering,
College of Electrical & Mechanical Engineering (CEME),
National University of Science & Technology (NUST)
Islamabad, Pakistan
shah.jahan18@ce.ceme.edu.pk
Kashif
Department of Computer & Software Engineering,
College of Electrical & Mechanical Engineering (CEME),
National University of Science & Technology (NUST)
Islamabad, Pakistan
muhammad.kashif18@ce.ceme.edu.pk
Muhammad Talha Riaz
Department of Computer & Software Engineering,
College of Electrical & Mechanical Engineering (CEME),
National University of Science & Technology (NUST)
Islamabad, Pakistan
talha.riaz18@ce.ceme.edu.pk
Muhammad Abbas
Department of Computer & Software Engineering,
College of Electrical & Mechanical Engineering (CEME),
National University of Science & Technology (NUST)
Islamabad, Pakistan
abbasamir@hotmail.com
ABSTRACT
A software must do what it intends to do. The quality is core
factor in IT Industry and in software products. The quality of the
product is main concern of the producer and the main requirement
of a customer. Software testing is the core activity of quality
assurance and very important phase of software development life
cycle. Quality of any system is known by testing that product.
With the advancement of IT industry many new verification and
validation technique are introduced over a time. New trends are
well adopted in developed countries like USA, UK etc. But not as
well in IT of developing countries like Pakistan. The need of this
paper is due to a large number of outsourced projects and also
multinational IT companies in Pakistan. Due to high demand of
outsourced projects the software testing become an integral part of
IT Industry of Pakistan. Our team has collected data about testing
in IT Industry of Pakistan which will be benefit for the testers in
IT Industry not just in Pakistan but worldwide. These results
could be use by researchers to identify the strengths and weakness
of testing in IT industry.
CCS CONCEPTS
Software and its engineering → Software testing and
debugging
KEYWORDS
Software Testing in Pakistan, Software testing practices, Software
testing, Software testing in developing country
1 Introduction
IT industry is the backbone of every industry and the software is
the backbone of IT industry. The failure of software can produce
catastrophic results. The cost of software failure is another
headache as in US only $62 billion were wasted due to software
failure. The cost of software failure is now more catastrophic as
the software is being used in real time, finance and health industry
where the effect of single error can have significant effects[1, 2].
Due to high use of software, the cost and effect of failure
increases, so there is a rise in software testing [3].
The software testing is very-crucial phase of software
development life cycle. Today IT industry acknowledge that
software testing is the core activity to produce defect free software
product[4]. Software testing is very wide fields and the focus of
research is on software testing as a generic activity and there is
need to explore this field in more depth. There is a difference
between literature and industry practices about software
testing[5].
Software testing are built upon domain experience gained in
industry; the software testing problems are technical. The
practices should be built upon rigorous and systemic approach
suggested by[6] who presents three case studies from industry
which shows that organizational structure of IT organization has
very significant effect on software testing and results[2]. Ahonen
and et al. [7] shows that the most accurate way of retrieving
software testing results is by observing the professional testers.
IT projects are complex due to involvement of hardware, network
and software. The need of software testing increased and also the
cost of testing. 90% of projects are outsourced so the cost of
software failure increases as it requires more money to fix the
errors in distributed projects[8].
To achieve global success in international market the Pakistani
companies must succeed the quality standards of leading software
organizations. Due to rapid change in world two fields are rising,
ECBS '19, September 2–3, 2019, Bucharest, Romania
© 2019 Copyright is held by the owner/author(s). Publication
rights licensed to ACM.
ACM ISBN 978-1-4503-7636-5/19/09…$15.00
https://doi.org/10.1145/3352700.3352724

web-enabled and mobile-enabled e-commerce. Software testing is
core for these fields due to fast turnout and short releases. The
customer demands higher quality standards. The gap between
where IT industry of Pakistan stands now and where should stand
and where could stand is very large. The above scenario describes
the perspective of Pakistani IT Industry but most of the other
developing nations are facing same problems in IT industry.
Every nation needs strong IT industry to succeed internationally.
A survey was conducted about software testing from the industry
and from the individual professionals. Online survey technique
was being used as Google Form and in some cases, interview was
conducted. This survey showed how mature Pakistani IT industry
is and how much organizations are paying attention to testing and
testers as well. This survey covers many aspects of software
testing from different testing-techniques from training of
individuals to education of testers. This survey will be equally
beneficial for the testing team, quality assurance team, researchers
and academia.
Rest of the paper is organized as follows. An extensive literature
review regarding survey of software testing is provided in Section
2. In Section 3 the survey results will be analyzed and discussed
and section 4 summarizes the findings and discusses the lessons
learned.
2 Literature Review
Performed very intensive survey of different key factors of
software quality in developing countries which consist upon 35
questions. This search exhibit that the products were not tested as
per test plan [9].
Torkar and et al. [10] Performed a similar survey about software
testing and reuse. There is a rise in reused code but testing of this
reused code is very limited. 52% of developers test their code,
more than 40% coder test most valued units. The open source and
business developers test their code equally. Developers like KISS
means keep it simple stupid whenever a new tool is released.
Geras and et al [11] Performed a survey in Alberta. The industry
in Alberta test less than the IT industry in United States. One of
the big causes of this less adoption of extreme programming. This
survey showed that most of the organizations produces error
prone test cases and a lack of criteria to stop the testing phase.
Due to this, these organizations release defective software
products. The most popular technique is Unit testing and there is
an increase in automated unit testing in development community.
Only 30% of the organizations uses automated testing with Junit,
NUnit or CPPUnit.
Kasurinen and et al. [12] Did an extensive industry study of 31
organizations. 55 industry specialists were interviewed about
software test automation. The testing process only receive three
fourth resources they required, and access of these resources is
limited. The adoption of automated testing tools is difficult in
industry. Even testing efficiency increases there are still 25%
effort dedicated to testing which is half of which is required. Only
26% of the participants use automated test cases. There is an
ambiguity that it depends upon that which organization is using
automation and how automation should be implemented.
Deak and et al. [13] Performed a research which was combination
of interviews and surveys with mix of qualitative and quantitative
questions. This research provides the current trends of organizing
software testing activities in IT industry. A new trend is that the
tester is part of the development team or a secondary role for
developer. The existence of testing department in small or
medium size organizations is nullable. Automated testing required
more budget and become complex that is why small automation is
being achieved.
Performed a nationwide survey about software testing in Canada.
This survey is the extension of surveys they performed in 2004
and 2009. This survey showed that
(I) There is an increase in training related to testing
(II) Unit testing and functional testing still the most
popular
(III) There is an increase in interest in mutation testing
(IV) Traditional test-last development has the most use
in IT industry of Canada[14-17].
Mailewa and et al [18] Purposed a novel testing technique for
testing of large set of test cases. A combinational testing. Test
automation is the solution of time problems in software testing.
Pairwise testing for combinational testing and selenium tool for
test automation of web projects. Also emphasize the advantages of
automation testing over manual testing.
The use of formal testing methods reduced even in real time
systems. Now organizations use more automation testing. The
cause of automated testing is high if there is use of agile software
development. They performed a survey of software testing and the
results showed that use of capability maturity models decreases
whilst automate testing become increasing common and more
sophisticated. Testing perform in design phase is decreasing while
acceptance testing increases[3].
Testing is one of the most important Phase of SDLC for quality
assurance but neglected as a result of time and budget. Sub-
optimizing activities of testing could lead to failure of software
projects. The testing should not only consider in testing phase of
SDLC (software development life cycle). The study found lack of
personal competences of testing is one of the major cause for
defected software product[19].
Performed an extensive study about stakeholder’s satisfaction and
advancement and penetration of agile methodologies in IT
industry of South Africa. This study based upon self-reported data
from 59 organizations. Ferreira and et al. [20] performed a survey
about unit testing of software process improvement network
(SPIN) which compress 50 representatives from different
companies. Runeson and et al. [21] a first time ever survey on the
subject of software testing in Australian ICT industry which
contains results collected from 65 organizations. These
organizations responded that they use at least one structured
software testing methodology. 65% of these organizations use
black-box testing while other use white-box testing. 65% of these
42 organizations carried out testing on 80% of their projects.[22]

Sneha and et al. [23]Performed and extensive research about
software testing methods and automated testing tools. This study
explores the testing techniques and tools and also automation
verses manual testing. The main goal of a testing activity is to
deliver software product with no or few errors. This study covered
unit testing, integration testing etc. Testing methodologies like
white-box testing, black-box testing etc.
With emergence of agile the need of testing increases. The
traditional methodologies has a separate phase for testing while in
agile it is very limited.[24]in agile factory model the defects
occurs are of severity of 3.26% of defects are defects during
acceptance integration and performance. While 17% in factory
acceptance testing[25]. The software automation test generation
can be modernized by agile development methods. This method
purpose team meetings of teams that develop the software product
and automated test software. The use of automation increases in
later sprints of Scrum which will speed up the delivery[26].
3 Methodology
The questioner-based survey methodology was used to get
responses from different IT organizations of Pakistan. We used
google forms, interviews and hard copy for this survey. In
interviews we asked questions of google form directly to the
participants. The GQM [27]approach was used in this survey. The
goal of this survey is to provide the details of the practices used
for software testing in IT Industry of Pakistan, to identify what
testing methods, tools and metrics are used by professionals and
researchers in Pakistan as well as worldwide. The goal of this
survey also is to identify the latest trends in software testing and
direction where we are going right now. This will identify the
strengths and weakness and also encourage the collaborations
between industry and academia. Following are the research
questions we raise in this research:
RQ 1: How much effort is spent on different levels of software
testing.
RQ 2: Which are the most and least used test technique.
RQ 3: What specific testing tools are used and how much
automation is used in software testing.
RQ 4: What type of test-metrics are used in Software testing.
RQ 5: How the teams and processes are managed during testing.
RQ 6: How much software testing training is provided by
organizations to their employees.
RQ 7: What are the challenges faced by industry.
Industry professionals and Authors did a peer-review on this set of
questions. This consultation was done to reduce the complexity in
form and to increase the similar terminology so that it could
match with action of respondents. As software testing terminology
in academia is slightly different from the industry where it is used
as (system testing, functional testing etc.). The feedback of
professionals was used to finalize the survey. Most of these
questions have quantitative predefined multiple-choice answers.
Few questions have qualitative answers.
The first category of questions (1-10) collected the information of
respondents and respondent’s organizations. The second section
(11-13) questions collected the response\es for test type/levels.
This section will show what type of testing (unit testing, system
testing etc.) is being perform in the industry and also shows which
organization are making the testing levels to identify the
characteristics of product under test. The third category (14-15)
questions showed what software testing technique are being used
and about a new and rising software testing technique which is
mutation testing.
In forth category (16-17) questions asked about automated testing
and tools/framework use for software automated testing. In fifth
category (18-19) we asked about how they measure software
testing and what metrics they use. In sixth category (20-22)
questions asked about the criteria used to terminate the testing
activities, what barriers prevent from adopting systematic testing
methodology and testing tools, and time spend on software
testing. In the seventh category (23) questions asked about the
software testing training and this question does not count for self-
study. And in eight and last category (24) questions asked about
involvement of organizations into any research and the three most
testing challenges they want academia to solve.
We send this form to 80 organizations and 450 individuals from
IT Industry of Pakistan.53 organizations while 185 individuals
participate in this survey the responses are 238 with 80% om
google form and 20% from interviewing and hardy copy. Many
organizations are CMMI and ISO ranked. These organizations are
large, medium and small sized. The responses from medium sized
and emerging companies is very satisfying.
3.1 DATA ANALYSIS & SURVEY RESULTS
3.1 Profile of participants
3.1.1 Respondents designation
The Figure-1 shows the position of the participant.This question
was a multiple-choice because a developer may be a tester at same
time and vice versa. Means one participants could hold more then
one position. People in diffrent positions see and rate the different
problems diiferently that is why this survey allow other positions
holders other than software tetsre are allowed to take part in this
survey.
Figure. 1. Respondents position
We analyzed the frequency in this question in which participants
holds more then one position. 4 reposndants were tester and
devleoper at same time while 4 people were bussiness analyst and
developer at same time. The ratio of joint position is rather low

3.8%. The highest joint position ware tester and QA Lead which
could be inter-related.
3.1.2 Participants Experience
Figure-2 shows the participants work experience in overall IT
industry and Figure-3 particularly in software testing. 15% and
5% have over 5-10 years of experience in software development
like developer, business analyst etc., respectively. 10% and 4%
have over 5-10 years’ experience in software testing. These results
are because when a newcomer enters in software industry, they
start working in software development phase and after experience
they enter in software testing phase.
Figure. 2. Working Experience
Figure. 3. Testing Experience
43% of the participants has 2-5 years of experience in overall
software development which is why more than the 26% of
participants are in their start of career who has 0 to 2 years of
experience of software development while 60% of the respondents
are in start of testing career. It shows more and more people are
choosing software testing as a career.
3.1.3 Organization Category
In this question we asked for the category of the participant’s
organization. We provide six choices and an option of others also.
But all the responses fall into one of the six provided categories.
Figure-4 shows the results.
Figure. 4. Organization Categories
Most of the responses are from software contractor, developing
customize software and developing software for internal use. The
implementation of testing activities is different for different
categories. The client of customize software has one firm while
package software used by many clients.
3.1.4 Academic Degree
To asked participants educational background, we asked for their
highest degree. Figure-5 shows the results
Figure. 5. Academic Degree
The results are very satisfying as 60% of the respondents holds
BS degree while 36% of the respondents have master’s degree
which is quite high ratio for a developing IT Industry. Just 1% of
the respondents have PHD degree as most of PHD’s prefer
academia as a profession.
3.1.5 University Degree
As a follow up of previous questions we asked the participants
about their major degree (software engineering, Computer
Engineering, computer science e.g.). the results are showed in
Figure-6
Figure. 6. University Degree

Software testing and computer science was the major degree of
most of the participants. 61% of the participants have software
engineering as their major while 29% have computer science and
8% have MBA as their major Subject. These business participants
hold position of business analyst.
3.1.6 Testing Certification
In this question we asked the participants about the testing
certification they have. The results are not very impressive
compare to develop countries but impressive for a developing
country. The Figure-7 displays the results.
Figure. 7. Testing Certification
51% of the participants take at least one type of testing certificate.
While 49% have no certificate. These are those people who was
developer and then turned to tester and have a lot of development
experience. The trend of certification could be seen in newcomers
who chose software testing as a profession. 19% of the participant
have Test Managed approach (TMap) certification while 12%
have Certified Associate in Software Testing (CAST). 9% each
have foundation level certificate tester (ISTQB) and Certified
Software Quality Engineer (CSQE).
3.1.7 Type of Methodology is being use
In this question we asked for the methodology is being used in
participant’s organization. As this is the era of Agile and most of
the participants belongs to medium level organizations while large
organization still use traditional methodology like waterfall and
iterative. Due to complexities of agile in large scale software.
Figure-8 shows the following results
Figure. 8. Current Development Team
47% of the respondents respond that they use agile methodology
while 28% of the participants are still using traditional
methodology. A new trend could be seen in small and medium
size organizations is 25% do not explicitly distinguish which
means there could be mix of both methodologies.
3.1.8 Organization Size
In this question we asked for company size of respondents. There
are five categories of companies
• Very small (1-09)
• Small (10-49)
• Medium (50-249)
• Large (250-999)
• Very large (1000+)
Figure. 9. Number of Employees
45% of the participants belong to medium size organizations
while 27% belongs to small companies. 9% each from large and
very large organizations. It showed that participants from all size
organization took part in this survey which is very satisfactory.
3.1.9 Programming Language
In this question we asked for the programming language use in
participant’s organization for development. This question is very
important as asp.net and java provides very strong automate
testing framework (JUNIT and NUNIT). The result is shown in
Figure-10
Figure. 10. Programming Languages
28% of the organizations use java because of latest trend of
mobile application development and android development. 26%
of the organization use C/C++ are being used in embedded
software, while 18% are uses .NET family (C#, VB) and at last
(28%) of the organization are using other languages like PHP,
Ruby on Rails etc.
3.2 Test Type/Levels
3.2.1 Test Case and Test Suits
In this question we asked about different types of test to the
participants. The results are shown in Figure-11 which are of very
changing as 30% of the participants use User acceptance testing.

27%-unit testing, 23% of the respondents use functional/system
testing. Only 5% of the participants use performance testing while
just 1% say they use all or mix of these testing types.
Figure. 11. Test Cases and Test Suites
3.2.2 Effort Spent on software Testing
In this question we asked about effort estimation spent on
different software testing types (Unit testing, System testing). A
participant could choose 40% on Unit testing, 40% on Functional
Testing and 20% on system Testing. The results are shown in
Figure-12
Figure 14 Generate Test Cases
Figure. 12. Effort spent on testing
Most of the respondents spent 21-40% on Unit testing while 41-
60% is spent on user acceptance testing. Those results are same
for System testing, GUI testing, Performance testing, Stress
testing and most of the participants 0-20% on other type of
testing.
3.2.3 Which Testing Activities
In this question we asked for the type/phase of SDLC. The results
are shown in Figure-13
Figure. 13. Test Activities
The results are varying as 36% of the respondents respond that
they use Test-Last development (TLD) while 35% of the
respondents use Test-Driven development (TDD). 27% of the
respondents uses Behavior-Driven development (BDD). While
1% using smoke or sanity testing.
3.3 Testing Techniques
3.3.1 Generate Test Case
In this question we ask for the technique participants use for
testcase generation. Figure-14 shows the results. These results are
mix as 29% of the respondents uses model-based technique. 23%
of the respondents uses boundary value analysis to generate test-
cases. 17% and 19% use code-coverage (white-box testing) and
exploratory testing respectively while 7% use category
partitioning (i.e. equivalence classing). 15% of the participants do
not use any technique explicitly.
3.3.2 Mutation Testing
In this question we asked about mutation testing (in which faults
are injected). We asked do they use mutation testing in their all
projects or in some projects not using it at all. The Figure-15
shown the results

Figure. 14. Mutation Testing
The results are very satisfying. As 26% of the respondents
respond that they use mutation testing in their all projects while
42% use it in some of their projects. 32% of the participants
mostly belong to small organizations do not use mutation testing.
3.4 Automatic Testing and Tools
3.4.1 Automatic Testing and Manual Testing
Figure. 15. Automate vs Manual Testing
In this question we asked to respondents about automated testing
and manual testing. Figure-16 shown the results. The results are
unexpected as the use of manual testing 21-40% while use of
automatic testing is 41-60% This is due to high use of Java and
.NET framework and web (selenium) and android development
which is help-full in automated testing.
3.4.2 Testing Tools
Figure. 16. Testing Tools
We asked about which software testing tools and testing
framework used by most of the participants. Figure-17 shown that
38% of the participant use family of commercial functional test
tools (for non-website) e.g. IBM. 26% each use testing tools for
web application and XUNIT framework e.g. JUNIT, NUNIT.
10% of the participants use tools and framework.
3.5 Software Testing Measurements
3.5.1 What type of Testing Measurement
Measurement in software testing is very important instrument. It
is very important to determine that what type of measurement are
done for software testing in Pakistan. As we measure these
measurements in white-box testing and in black-box testing we
use code coverage to measure how much of the system is tested.
The choice given to participants was
• Line(statement) coverage
• Decision(branch) coverage
• Condition coverage
• MC/DC coverage
Figure. 17. Code (Test) Coverage
Figure-18 showed that 99% of the participants use one of the
above approaches. 43% of the respondents use decision (Branch)
coverage while 37% of the respondents use condition coverage.
15% uses Line (Statement) coverage and 45 of MC/DC coverage.
3.5.2 Test and Quality Metrics used in projects
We also asked the participants about test and quality metrics they
use in current or previous projects. Figure-19 shown the results
Figure. 18. Test and Quality Metrics
39% of the respondents say they use number of test-cases
execution defined by time while 20% use total number of defects
detected per day (week, month). 16% and 15% respondents
respond that they use testers defect (bug) detection productivity
(bugs found per day by each tester) respectively. 10% uses
number of passing user acceptance tests.
3.6 Termination and Prevention
3.6.1 Criteria to Terminate Testing Phase
In this question we asked the participants that what criteria they
use to terminate the testing phase. This question is very important
as it tells how to terminate a testing phase. It directly depends

upon the quality of product and size of company. Figure-20
presents the results
Figure. 19. Termination Criteria
35% of the respondents responds that they execute all test cases
without finding more defects(bugs). 22% and 21% of the
respondents say they use fixed time duration and no bugs are
found anymore (reliability growth/saturation models). Other
participants use some other criteria.
3.6.2 Hurdles in Adopting Systematic Software Testing
In this question we asked the respondents about the barriers that
prevent them adopting systematic software testing methodology.
Figure-21 shows that 62% of the participants respond that time is
the most effective barrier while 22% responds that no support of
higher-level management and 9% say cost is the barrier while
adopting the systematic testing methodologies.
Figure. 20. What prevents from adopting Testing
3.6.3 Effort spent on overall Testing
In modern software development, testing phase is taking more and
more effort in shape of budget and time. It is very important to
know how much Effort is being spent on software testing activity
in IT industry of Pakistan. Figure-22 presents following results
Figure. 21. Effort spent on overall testing
46% of the participants spent 41-60% effort on software testing
phase. 19% and 17% of participants spent 21-40% and 61-80% of
budget and time on testing phase respectively. Just 12% of the
participants spend below 20% of their budget and time on testing.
3.7 Training and Challenges
3.7.1 How much Formal Training
The formal training about testing provides a solid foundation for
software testers. It is very important to have formal training for
software testing. This training includes on-site training and
training in university. This question excludes the self-training and
study in this survey. This training is in hours. Figure-23 shown the
results
Figure. 22. Formal Training
32% of the participants have 41-60 hours of training while 21%
have 21-40 hours training. 17% of the participants have 61-80 and
0-20 hours of formal training each. 6% and 7% of participants
have 81-100 or 0 hours of training respectively.
3.7.2 Testing Challenges
In this question we asked the participants for their three top
challenges they want academia to solve. These are the challenges
faced by the industry professionals. Figure-24 shown the results
Figure. 24. Testing Challenges
46% of the participants faces time, cost and effort and staffing.
46% of the respondents respond test-code engineering as the
biggest challenges they are facing. While 42% of the participants
think training is the problem. While 22% think metrics is the
problem they are facing, and they want academia to solve these
problems.

4 Discussion
Figure. 23. Overall Results
This research paper shows that most of the testing is performed by
developers. Most of the developers has BS Software Engineering
as academic degree. Mostly software organizations are software
contracting companies. The IT organizations has most employee
with average experience of 02-05 years with 0-2 years of
experience in Software testing. The software companies are using
agile methodology. The average size of companies in Pakistan is
50-249 employees. C++ and Java still dominate the Pakistani IT
Industry. User acceptance is the most used metric for product
quality. 41-60% of effort is spent on different testing techniques.
Most of the organizations use test-last development and use
model-based technique for generating Test-cases. A new trend is
being seen is use of Mutation testing in IT industry of Pakistan.
Some are using it in all projects but mostly use it in some projects.
Design (branch) coverage is used for code coverage and number
of test cases in defined period is the quality metrics. The
termination criteria of Testing Phase are all test cases executed
within a time. This paper shows that time is barriers in choosing
systematic testing methodology. This paper presents a new trend
in IT industry of Pakistan which is the formal training of
employee on software testing and the top three challenges faced
by IT industry which are Triple constraints and, Test-code
Engineering and Training. This study showed the overall growth
of IT industry of Pakistan especially in Software Testing.
There are many research papers about software testing and quality
but very few related to software testing in IT industry of Pakistan.
Some research is being done on topic of software quality in
Pakistan which covered software testing to some extent. We try to
cover every aspect of testing in IT industry of Pakistan. Due to
nature of our survey we could not compare our set of results with
other previous researches on the topic of Software testing related
to IT industry of Pakistan but could compare some question with
similar studies in other countries like Canada, Australia and
Norway.
There is some Common trend in IT industry of Pakistan like other
countries:
• Pakistan IT industry is dominated by Black-box Testing
which is like the finding in Canada reported in Vahid
Garousia and et at. (2012).
• Still decision and condition coverage are being use for
code coverage in Pakistan similar to finding of Vahid
Garousia and et at. (2012).
• User Acceptance testing is the most used testing
technique than unit and funxti0onal testing in Pakistan
which is similar to finding of Anca Deak and et al.
(2013) in Norway.
• A new trend of spending of 41-60% of effort on testing
is similar to finding of Taipale and et al. (2005)
• .A similar trend could be seen in Pakistan is the formal
training of testers which is 93% more than the findings
of S.P. Ng et al[23] which are 72%.
Some of our results differ then the finding in other countries
and are as follows:
• Automated testing is new trend in Pakistan’s IT industry
due to high development of Mobile and Web
development.
• IT industry of Pakistan is paying more and more
intention to mutation testing which varies then all three
countries mentioned in this paper.
• Junit is still dominate the IT industry with an increase of
use of Web testing tool (Selenium) with shows a
difference with the findings of Taipale et al. (2005) and
Vahid Garousia and et at. (2012).
• Testing certification is now become a core requirement
for newcomers who chose software testing as their
career which is similar to other countries.
• A new trend of test-driven development inn Pakistan but
still Test-last development dominate the industry. These
finding are similar finding of Vahid Garousia and et at.
(2012) [14] and T. Hynninen et al. [3]
• Time is the main barrier while choosing the systematic
testing methodologies which clearly differs from S.P.
Ng et al[23]

5 Conclusion and Future Work
In this paper, we have conducted a survey among organizations of
Pakistan which are paying more attention to the software testing
activity. Software testing is the core activity of software SDLC. In
this survey we witnessed some encouraging trends in Pakistan.
Most of the testers start their career as a developer and when they
got experience, they move to the job of software testing. Most of
the testers are well educated and have received relative degree
with IT. Industry is paying attention to Test-Certification. Effort
spending on testing is better than some developed countries.
Diversity of IT industry is the key to success like in Canada most
of the IT industry use TLD but in Pakistani industry is using TLD,
TDD and BDLL. organizations are using systematic testing
methodology. Selenium is tool use for automating testing of web
development that’s why Automate testing is getting more and
more attention in Pakistan. Mutation testing is relatively new and
not many developed IT Industry is still using it but in Pakistan we
witnessed a different trend as most of the organizations are using
Mutation testing in some part of testing. There were times when
formal training was conducted on Testers end but now most of the
organization are conducted Formal training for their testers but
still the training time is very low. Testing phase termination
criteria is still time, budget or manager. In large organization there
are formal methods for termination but in medium and small
organization still the manager’s orders are the criteria.
This paper showed the growth and trends in IT industry of
Pakistan with respect to some developed countries like USA,
Canada etc. but also alarm us on the growth speed. Pakistan still
lagging behind where it could be at this time but with passage of
time and more and more investment in IT sector and shifting of
multinational companies in Pakistan, we hope the IT industry of
Pakistan will flourish in near future and will challenge not just
other developing industries but also many developed IT
industries. There is need to do more surveys in Pakistan as well as
other developing countries and compare these results with them,
so we could assess where IT industry of Pakistan stands now and
where it should be in future. In future we will provide the
relationship between different questions to assess more accurate
and related results.
REFERENCES
1. Kit, E., Software testing in the real world: improving the process. 1995:
Addison-wesley.
2. Planning, S., The economic impacts of inadequate infrastructure for software
testing. National Institute of Standards and Technology, 2002.
3. Hynninen, T., et al. Software testing: Survey of the industry practices. in 2018
41st International Convention on Information and Communication Technology,
Electronics and Microelectronics (MIPRO). 2018. IEEE.
4. Bertolino, A. Software testing research: Achievements, challenges, dreams. in
2007 Future of Software Engineering. 2007. IEEE Computer Society.
5. Rooksby, J., M. Rouncefield, and I. Sommerville, Testing in the wild: The
social and organisational dimensions of real world practice. Computer
Supported Cooperative Work (CSCW), 2009. 18(5-6): p. 559.
6. Engström, E. and P. Runeson. A qualitative survey of regression testing
practices. in International Conference on Product Focused Software Process
Improvement. 2010. Springer.
7. Ahonen, J.J., T. Junttila, and M. Sakkinen, Impacts of the organizational model
on testing: Three industrial cases. Empirical So ftware Engineering, 2004. 9(4):
p. 275-296.
8. Batra, D., Modified agile practices for outsourced software projects.
Communications of the ACM, 2009. 52(9): p. 143-148.
9. Naqvi, N., A. Khan, and A. Khan, Survey to Improve Software Quality
Assurance in Developing Countries. International Journal of Technology and
Research, 2015. 3(1): p. 1.
10. Torkar, R. and S. Mankefors. A survey on testing and reuse. in Proceedings
2003 Symposium on Security and Privacy. 2003. IEEE.
11. Geras, A.M., M. Smith, and J. Miller, A survey of software testing practices in
Alberta. Canadian Journal of Electrical and Computer Engineering, 2004.
29(3): p. 183-191.
12. Kasurinen, J., O. Taipale, and K. Smolander, Software test automation in
practice: empirical observations. Advances in Software Engineering, 2010.
2010.
13. Deak, A. and T. Stålhane. Organization of testing activities in Norwegian
Software Companies. in 2013 IEEE Sixth International Conference on Software
Testing, Verification and Validation Workshops. 2013. IEEE.
14. Garousi, V. and J. Zhi, A survey of software testing practices in Canada.
Journal of Systems and Software, 2013. 86(5): p. 1354-1376.
15. Garousi, V., et al., Testing embedded software: A survey of the literature.
Information and Software Technology, 2018.
16. Garousi, V. and T. Varma, A replicated survey of software testing practices in
the Canadian province of Alberta: What has changed from 2004 to 2009?
Journal of Systems and Software, 2010. 83(11): p. 2251-2262.
17. Garousi, V., M. Felderer, and F.N. Kılıçaslan, A survey on software testability.
Information and Software Technology, 2018.
18. Mailewa, A., J. Herath, and S. Herath. A Survey of Effective and Efficient
Software Testing. in The Midwest Instruction and Computing Symposium.
Retrieved from http://www. micsymposium.
org/mics2015/ProceedingsMICS_2015/Mailewa_2D1_41. pdf. 2015.
19. Belt, P., et al., Improving the efficiency of verification and validation.
International Journal of Services and Standards, 2008. 4(2): p. 150.
20. Ferreira, C. and J. Cohen. Agile systems development and stakeholder
satisfaction: a South African empirical study. in Proceedings of the 2008
annual research conference of the South African Institute of Computer
Scientists and Information Technologists on IT research in developing
countries: riding the wave of technology. 2008. ACM.
21. Runeson, P., A survey of unit testing practices. IEEE software, 2006. 23(4): p.
22-29.
22. Ng, S., et al. A preliminary survey on software testing practices in Australia. in
2004 Australian Software Engineering Conference. Proceedings. 2004. IEEE.
23. Sneha, K. and G.M. Malle. Research on software testing techniques and
software automation testing tools. in 2017 International Conference on Energy,
Communication, Data Analytics and Soft Computing (ICECDS). 2017. IEEE.
24. Agarwal, A., N. Garg, and A. Jain. Quality assurance for Product development
using Agile. in 2014 International Conference on Reliability Optimization and
Information Technology (ICROIT). 2014. IEEE.
25. Doherty, B., et al. Defect Analysis in Large Scale Agile Development: Quality
in the Agile Factory Model. in 2016 Joint Conference of the International
Workshop on Software Measurement and the International Conference on
Software Process and Product Measurement (IWSM-MENSURA). 2016. IEEE.
26. Sultania, A.K. Developing software product and test automation software using
Agile methodology. in Proceedings of the 2015 Third International Conference
on Computer, Communication, Control and Information Technology (C3IT).
2015. IEEE.
27. Mandić, V. and N. Gvozdenović, An extension of the GQM+ Strategies
approach with formal causal reasoning. Information and Software Technology,
2017. 88: p. 127-147.