Drastically Improve Test Coverage in Agile Projects through Crowdsourced Exploratory Testing

Get a Quote

Drastically Improve Test Coverage in Agile Projects through Crowdsourced Exploratory Testing

By Bala Kalimuthu, National Manager, Digital Solutions.

Download the full article here.

Agile software development is continually gaining popularity for the speed and efficiency it brings to projects, but the integration of testing presents several challenges. To understand how it fits the Agile landscape, we must first understand Agile itself.

In the context of software development, ‘Agile’ refers to a specific set of principles outlined in The Agile Manifesto. These principles advocate a fast and flexible approach to projects with an emphasis on early delivery, evolutionary development, continuous improvement, automation and adaptive planning.

Projects under an Agile framework typically require software deliveries every 2 to 4 weeks as part of numerous phases known as ‘iterations’. When using the scrum methodology – a common application of the Agile framework co-founded by Dr. Jeff Sutherland – each iteration is referred to as a ‘sprint’.

The nature of Agile projects means all testing must be completed within demanding timeframes, which can mean only the most critical and high-risk ‘happy path’ scenarios are tested when budget and resource constraints are considered. Testing environments also offer limited coverage, especially for highly user-centric software.

Utilising the crowd
Crowdsourced testing is a great fit for Agile software development: it allows the sourcing of hundreds of local or international testers within a few hours, with the ability to test for compatibility, functionality, accessibility, user experience and much more.

The major benefits include:
• Shorter time-to-market
• Increased conversions
• Improved coverage
• Real users with real devices
• Better defect detection
• Higher cost efficiency

Who is in the crowd?
Every project has unique requirements, and crowd testing can be tailored as appropriate. Agile teams can complete the necessary tests within a small time frame by sourcing specialist testers, seasoned industry professionals and subject matter experts, recent graduates, customers, staff or even the general public. These testers may be onshore or overseas, and can be dispersed, partially dispersed, co-located or in distributed sub-teams.

Leveraging crowdsourced exploratory testing
Crowdsourced exploratory testing expands coverage by taking a more hands-on approach: instead of each tester following strict, step-by-step instructions, they are asked to execute a test or achieve a desired outcome through their own means. The approach is so called for its emphasis on exploring the software with the freedom to rely on experience, impulse and creativity to create new tests along the way.

In order to minimise production issues, Agile projects need to test with maximum efficiency and effectiveness.

Crowdsourced exploratory testing opens numerous innovative pathways through which to achieve this goal, with the ability to dramatically increase test coverage within two to three days for a fraction of the traditional cost. Major enterprises are able to leverage this approach to identify hundreds of bugs overnight – AMP, Village Cinemas and Simplot being notable examples in recent years.

Crowdsourced exploratory testing can be applied to Agile projects in numerous ways depending on the needs of the company and the nature of each project.

The following options are the most common forms of integration based on a number variables:

Option 1: Crowd test 3 to 4 weeks before a major release

Crowd test 3 to 4 weeks before a major release

In the first and most basic pathway, projects still rely on traditional in-house testing of each product during its development. After the final iteration and prior to release, the product is crowd tested in an effort to identify any defects that were missed during in-house testing. This may be implemented a few weeks before release to allow time for rectification, and enables the delivery of higher quality software when compared with products only tested internally.

Options 2: Crowd test after 3 or 4 sprints (Technical Spike)

Crowd test after 3 or 4 sprints (Technical Spike)

The second option available for Agile projects is to test after a group of ‘sprints’, which is another term for iteration used by those following the scrum method. With this approach, the crowd is engaged for testing after two or three sprints of development before each sprint is repeated as many times as necessary. The number of sprints completed before testing varies between projects, but the overall concept focuses on the advantages of crowd testing before the conclusion of the development phase: engaging the crowd earlier allows for more types of testing and more comprehensive coverage.

Option 3: Crowd test every sprint (1 out of 4 weeks)

Crowd test every sprint - 1 out of 4 weeks

Option 3 takes things a step further by implementing a phase of crowd testing at the end of every sprint. As an example, a typical Agile project may have a sprint duration of four weeks. The team would work on analysis, design and development from the start of Week 1 right through to Wednesday of Week 3, at which point the crowd starts testing and continues until the Sunday of Week 3 for a total of 5 days. This leaves time in Week 4 for bug fixing and retests, with the cycle repeating as part of a new sprint the following week. Sprint duration can vary based on the technology involved, with some stretching as long as two months.

Option 4: Crowd test every sprint (on the weekend)

Crowd test every sprint - on the weekend

Just as Agile projects can involve long sprints, they can also keep them quite short. For those that run sprints in around two weeks, option 3 is less viable due to the narrow time constraints. A common tactic to address this issue is to apply a similar philosophy but alter it to fit within a weekly schedule. For example, a two-week sprint would involve development from Monday to Friday with crowd testing starting at COB Friday and continuing until Monday morning.

Option 5: Crowd test every sprint (overnight)

Crowd test every sprint - overnight

Even still, option 4 does not always suffice. Some Agile projects run short enough sprints that even further integration is required. This pathway will usually involve development from 9am to 5pm and testing from 5pm to 1am, followed by validation from 1am to 9am and bug fixing from 9am onwards.

Option 6: Crowd test every sprint (overnight and weekends)

Crowd test every sprint - over night and weekends

The final option in our list of common Agile testing methods offers the highest level of integration. Although it requires enhanced communication and use of a digital crowdsourced testing platform is essential, it maximises collaboration by bringing the roles of developers and crowd testers closer together.

The value of integration

While the concept of combining crowdsourced testing with the scrum methodology is relatively young, it’s becoming a necessity for tackling Agile projects under climbing industry demands. In addition to shorter delivery times and coverage concerns, many businesses develop software with teams in multiple locations, with communication becoming more difficult as a result.

Crowdsourced exploratory testing is helping Agile companies adapt to this environment by enhancing collaboration and ensuring compatibility with sprint cycles. Through crowd testing platforms, testers are digitally connected with developers and each other to maximise efficiency and effectiveness. Altering an existing framework to integrate this approach can seem daunting, but it’s an invaluable change of mindset that causes dramatic improvements in product delivery.


Related Articles:

Creating Value with Agile Performance Testing


If your business is looking to adopt Agile methodology or already has, Revolution IT has a large team of specialist Agile practitioners you can leverage to help deliver on your projects.  

Not sure about crowdsourced testing? Never used the crowd before? Have reservations about its effectiveness as an alternative to traditional testing? Contact us for a demo.


Other Articles You May Also Like

A Day in the Life of a CrowdTester (aka The Busy Mum)

In my last blog, I discussed the different backgrounds of our freelanc... more

A Day in the Life of a Crowdtester (aka the contractor in between jobs)

As promised in our ‘daily life’ blog featuring the busy mum, we’re bac... more

Get in touch with us today. Send us a quick message.

*for employment enquiries please contact us via our Careers page