Challenges in Agile Testing

Every software development lifecycle model, from sequential to spiral to incremental to Agile, has testing implications. Some of these implications ease the testing process and some of these implications challenge testing.

A blended testing strategy consists of three types of test strategies:

• Analytical risk-based testing;

• Automated regression testing;

• Reactive testing (also referred to as dynamic

testing).

The blended testing strategy aligns well with Scrum and other agile methodologies.

In some cases, these strategies mitigate the testing risks and reduce the testing challenges associated with these methodologies.  

However, it does not resolve all of the risks and challenges, few of the challenges faced and the best approach to mitigate them have been discussed below:

Dealing with the Volume and Speed of Change

“Welcome changing requirements” is considered as one of the principles of agile development,

Many testing strategies, especially analytical requirements-based testing, become quite inefficient in such situations. 

However, risk-based testing accommodates change, since we can always add risks, remove risks, change risks, and adjust the level of risk.If test execution is underway, we can adjust our plan for the remaining period based on this new view of quality risk.

The reactive testing, not requiring much documentation, is also quite resilient in the face of change.

However, challenges arising from change in the definition of the product and its correct behavior, when the test team is not kept informed of these changes, or when the rate of change is very high, can impose inefficiencies on the development, execution, and maintenance of tests.

Remaining Effective during Very Short Iterations

Agile methodologies like Scrum are less formal and faster moving “sprint”’. These methodologies use short, fast-paced iterations, further squeezing the test

team’s ability to develop and maintain test systems, compounding the effects of change.

Testing strategies that include an automation element have proven particularly/sensitive to this challenge.

The risk-based element of the recommended strategy can help.

Risk-based testing focuses on the important areas of test coverage, and de-emphasizes or even cuts less important areas, relieving some of the pressure created by the short iterations. This ability to focus proves especially helpful for test teams also under tight resources constraints. Test teams in an agile world should develop, maintain, and execute tests in risk priority order.

Using risk priority to sequence development and maintenance efforts allows the test team to have the most important tests ready at the beginning of each sprint’s test execution period.

Receiving Code after Inconsistent and Often Inadequate Unit Testing

The short test execution periods on Agile sprints, compared to sequential projects, means that the degree of damage caused by one or two days of test progress blockage due to highly buggy code is higher than in a sequential project.

Delivery of unstable, buggy code will undermine one of the key benefits of the risk-based

testing portion of the recommended strategy, which is the discovery of the most important

defects early in the test execution period.

It also inevitably leads to a large degree of code churn during testing, since so much must change to fix the bugs.

The amount of change can ultimately outstrip even the ability of the best automated regression test system to keep up, which would then lead to lower defect detection effectiveness for the test team.

Managing the Increased Regression Risk

Agile methodology advocates emphasize good automated unit testing

in part to manage the regression risk inherent in such churn.

However, good unit testing has limited defect removal effectiveness.

Therefore, we need effective regression testing at the system test level (which has a higher level of defect detection effectiveness).

By combining risk-based testing with the automated regression testing, test teams can effectively manage the increased regression risk.

Making Do with Poor, Changing, and Missing Test Oracles

Agile methodologies de-value written documentation. Special scorn is reserved for specifications.

For example, the Agile Manifesto suggests people should value “working software over comprehensive documentation.”

This creates real challenges for a test team.Testers use requirements specifications and other documents as test oracles; i.e., as the means to determine correct behavior under a given test condition.

Testers in Agile situations are often given documents with insufficient detail, or, in some cases, given no such documents at all.No known test strategy can resolve this challenge. Resolving the challenge requires change management.

Further, the inability to determine precisely whether a test failed affects both the efficiency of the testing and the defect detection effectiveness.

When testers spend time isolating situations that the project team ultimately chooses to define as correct behavior, that takes away time they could have spent finding and reporting real bugs.

These bugs create subsequent problems customers, users, and technical support staff, and distractions for developers and test teams.

Further, the situation creates frustration for the testers that reduces their morale and, consequently, their effectiveness.

Testers want to produce valid information. When much of the information they produce – in the form of rejected bugs reports – ends up in the figurative wastebasket of the project that tends to make people wonder why they bother.

It’s important to realize that this reduction in test effectiveness, efficiency, and morale is a

potential side-effect of agile methodologies. Bad problems can get much worse when the test team is held accountable for outcomes beyond their control.

Holding to Arbitrary Sprint Durations

Some of our clients following agile methodologies like Scrum tend to ritualize some of

the rules of the process. The time deadlines for sprints seem particularly subject to this ritualization.

So, on the last Friday of the sprint, with development ending late for the sprint, the arbitrary deadline remains intact at the expense of the test team’s weekend.

Fully resolving this challenge requires team and management maturity. When people habitually and systematically over-commit,

If fixing the software estimation problems cannot occur, risk-based testing helps the test team deal with systematic and constant over-commitment.

To start with, when the test team is time-crunched over and over again at sprint’s end, the test team should accept that the project team’s priorities are schedule-driven, not quality-driven.

The test team should revise the risk analysis approach to institute an across the-

Board  reduction in the extent of testing assigned to each quality risk items during risk

analysis for subsequent projects.

That way, at least the test team won’t over-commit.

In some cases, in spite of reducing the scope of testing, the test team still can’t execute all the tests in the available time at the end of a sprint. If so, rather than ruining their weekend to run every test, the test team can select the most important tests using the risk priority number.

Less important tests can slip into the next sprint.

Conclusion

The test strategies recommended support the stated goals of Agile methodologies.

Risk-based testing supports increased quality, since it focuses testing on high-risk

areas where testing can significantly reduce the risk. Risk-based testing supports increased productivity, since it reduces or eliminates testing where the quality risk is lower. Risk based testing supports flexibility, since it allows regular revision of the quality risk items which re-aligns the remaining testing with the new risks and their new levels of risk.

Automated regression testing helps to contain the regression risks associated with Agile methodologies, allowing a higher rate of change.

Reactive testing allows testers to explore various aspects of the system that risk-based testing and automated regression testing together might miss.