Every software development lifecycle
model, from sequential to spiral to incremental to Agile, has testing implications.
Some of these implications ease the testing process and some of these
implications challenge testing.
A blended testing strategy consists of
three types of test strategies:
• Analytical risk-based testing;
• Automated regression testing;
• Reactive testing (also referred to as
dynamic
testing).
The blended testing strategy aligns
well with Scrum and other agile methodologies.
In some cases, these strategies
mitigate the testing risks and reduce the testing challenges associated with
these methodologies.
However, it does not resolve all of the
risks and challenges, few of the challenges faced and the best approach to
mitigate them have been discussed below:
Dealing with the Volume and Speed of
Change
“Welcome changing requirements” is
considered as one of the principles of agile development,
Many testing strategies, especially
analytical requirements-based testing, become quite inefficient in such
situations.
However, risk-based testing
accommodates change, since we can always add risks, remove risks, change risks,
and adjust the level of risk.If test execution is underway, we can adjust our
plan for the remaining period based on this new view of quality risk.
The reactive testing, not requiring
much documentation, is also quite resilient in the face of change.
However, challenges arising from change
in the definition of the product and its correct behavior, when the test team is
not kept informed of these changes, or when the rate of change is very high,
can impose inefficiencies on the development, execution, and maintenance of
tests.
Remaining Effective during Very Short
Iterations
Agile methodologies like Scrum are less
formal and faster moving “sprint”’. These methodologies use short,
fast-paced iterations, further squeezing the test
team’s ability to develop and maintain
test systems, compounding the effects of change.
Testing strategies that include an automation
element have proven particularly/sensitive to this challenge.
The risk-based element of the
recommended strategy can help.
Risk-based testing focuses on the
important areas of test coverage, and de-emphasizes or even cuts less important
areas, relieving some of the pressure created by the short iterations. This
ability to focus proves especially helpful for test teams also under tight
resources constraints. Test teams in an agile world should develop, maintain,
and execute tests in risk priority order.
Using risk priority to sequence
development and maintenance efforts allows the test team to have the most
important tests ready at the beginning of each sprint’s test execution period.
Receiving Code after Inconsistent and
Often Inadequate Unit Testing
The short test execution periods on
Agile sprints, compared to sequential projects, means that the degree of damage
caused by one or two days of test progress blockage due to highly buggy code is
higher than in a sequential project.
Delivery of unstable, buggy code will
undermine one of the key benefits of the risk-based
testing portion of the recommended
strategy, which is the discovery of the most important
defects early in the test execution
period.
It also inevitably leads to a large
degree of code churn during testing, since so much must change to fix the bugs.
The amount of change can ultimately
outstrip even the ability of the best automated regression test system to keep up,
which would then lead to lower defect detection effectiveness for the test
team.
Managing the Increased Regression Risk
Agile methodology advocates emphasize
good automated unit testing
in part to manage the regression risk
inherent in such churn.
However, good unit testing has limited
defect removal effectiveness.
Therefore, we need effective regression
testing at the system test level (which has a higher level of defect detection
effectiveness).
By combining risk-based testing with
the automated regression testing, test teams can effectively manage the
increased regression risk.
Making Do with Poor, Changing, and Missing
Test Oracles
Agile methodologies de-value written
documentation. Special scorn is reserved for specifications.
For example, the Agile Manifesto suggests
people should value “working software over comprehensive documentation.”
This creates real challenges for a test
team.Testers use requirements specifications and other documents as test
oracles; i.e., as the means to determine correct behavior under a given
test condition.
Testers in Agile situations are often
given documents with insufficient detail, or, in some cases, given no such documents
at all.No known test strategy can resolve this challenge. Resolving the
challenge requires change management.
Further, the inability to determine
precisely whether a test failed affects both the efficiency of the testing and
the defect detection effectiveness.
When testers spend time isolating
situations that the project team ultimately chooses to define as correct behavior,
that takes away time they could have spent finding and reporting real bugs.
These bugs create subsequent problems
customers, users, and technical support staff, and distractions for developers
and test teams.
Further, the situation creates
frustration for the testers that reduces their morale and, consequently, their
effectiveness.
Testers want to produce valid
information. When much of the information they produce – in the form of
rejected bugs reports – ends up in the figurative wastebasket of the project
that tends to make people wonder why they bother.
It’s important to realize that this
reduction in test effectiveness, efficiency, and morale is a
potential side-effect of agile methodologies.
Bad problems can get much worse when the test team is held accountable for
outcomes beyond their control.
Holding to Arbitrary Sprint Durations
Some of our clients following agile
methodologies like Scrum tend to ritualize some of
the rules of the process. The time
deadlines for sprints seem particularly subject to this ritualization.
So, on the last Friday of the sprint,
with development ending late for the sprint, the arbitrary deadline remains
intact at the expense of the test team’s weekend.
Fully resolving this challenge requires
team and management maturity. When people habitually and systematically
over-commit,
If fixing the software estimation
problems cannot occur, risk-based testing helps the test team deal with
systematic and constant over-commitment.
To start with, when the test team is time-crunched
over and over again at sprint’s end, the test team should accept that the
project team’s priorities are schedule-driven, not quality-driven.
The test team should revise the risk
analysis approach to institute an across the-
Board reduction in the extent of testing assigned to
each quality risk items during risk
analysis for subsequent projects.
That way, at least the test team won’t
over-commit.
In some cases, in spite of reducing the
scope of testing, the test team still can’t execute all the tests in the
available time at the end of a sprint. If so, rather than ruining their weekend
to run every test, the test team can select the most important tests using the
risk priority number.
Less important tests can slip into the
next sprint.
Conclusion
The test strategies recommended support
the stated goals of Agile methodologies.
Risk-based testing supports increased quality,
since it focuses testing on high-risk
areas where testing can significantly
reduce the risk. Risk-based testing supports increased productivity, since it
reduces or eliminates testing where the quality risk is lower. Risk based testing
supports flexibility, since it allows regular revision of the quality risk
items which re-aligns the remaining testing with the new risks and their new
levels of risk.
Automated regression testing helps to
contain the regression risks associated with Agile methodologies, allowing a
higher rate of change.
Reactive testing allows testers to
explore various aspects of the system that risk-based testing and automated
regression testing together might miss.
