Crowdtesting with Testeum!

Very often, when we develop a new application or a new feature of an existing product, we set up Quality processes to ensure that the implementation meets the needs of our clients: writing specifications, writing source code using BDD and TDD methods, functional tests (which can be automated), etc…

Then, when everything is ready, we often ask a group of people to “test” the beta version, the pre-prod environment, or other. These can be testers, developers, business consultants, PO, or simply “non-technical” people from our close circle. In fact, it is often these same people who are often asked to provide feedback as soon as new features are released, so that they are the first to provide feedback.  

What if Crowdtesting offered us a different angle of attack to more effectively identify bugs in our applications? This is what we will talk about in this article, thanks to feedback from Testeum, a solution developed by HighTest and dedicated to Crowdtesting, freshly released in 2020!

What is Crowdtesting ? 

This involves calling on a community oftesters to carry out aseries of checks on an application, identify any bugs or problems encountered, and report them rigorously to help the support teams reproduce them   .

This approach has several advantages: 

• Flexibility: the testers participating in your campaign will start the test phases in a very short period of time, each independently of the others.
• A fresh look: By dint of repetition, certain exploratory tests can be biased because the testers have acquired several reflexes on the use of the platform.
• A multicultural and multi-device approach: the diversity of tester profiles throughout the world, their digital application usage habits, but also the diversity of the devices used are significant assets for your tests, especially if you are targeting an international market.

Testeum is therefore a Crowdtesting platform which allows:  

• Publishers such as ProMyze to solicit professional testers to explore a range of product features.
• Testers from the general public have the opportunity to earn additional income, at the timing of their choice, and to discover new digital horizons with each session.

Define a test campaign with Testeum

Let’s now get down to business! First of all, we have started by creating a application, where the main focus is on how testers will be able to access it, and which are the target environments  .

Here, we would like ourThemisto be tested on MacOS and Windows, and under the Google Chrome browser. Next, Testeum asks us to import test cases from an Excel (.xls/.xlsx) file, which contains a set ofscenarios

,milestones and expected results. The structuring is trivial, and the scriptwriting is done without difficulty. Moreover, in some cases, the teams will have already defined functional test cases, so it will suffice to take them over to adapt them to the format desired by Testeum.  

Once imported, Testeum displays our test scenarios, so that we can detect any anomalies in their construction. In our case we had 10 modules and 19 test cases. If everything is OK, we can validate and move on to the next step!

The last step is to define which test environmentsthe testers will need to explore the application in:  .

• Free: the tester chooses the environment on which to carry out his tests, with no restrictions imposed by the publisher;
• Targeted : the tester chooses its environment from a list defined by the editor. This is what makes sense to select if there are environments that you expressly do not want to target ;  
• All: all environments selected by the editor will be tested  

We are now ready to validate the order, and it’s now up to the Crowdtesters to play!

Bug exploration and analysis

After a few days, we could explore the first results proposed by Testeum. The progress of the test cases is available, under a Burndown Chart approach. In any case, 5 Crowdtesters participated with a result of 95 executed test cases (19 test cases * 5 testers). We see below the distribution of the 34 Bugs reported according to severity level :

We can then move on to the bug exploration phase on a case-by-case basis. Each bug is precisely detailed (what was the environment), and is categorised according to the nature of the problem (e.g. display bug)    

We can then open each bug to find out more. The example below is trivial, and highlights a spelling mistake in the text :  

All bugs are complemented by screenshots andcomments.and the associated test case is also mentioned. For our team, it was quite simple to reproduce the bugs we encountered! The list of bugs can be filtered according to different criteria, notably OS and criticality, in order to prioritise the actions to be taken by the technical teams.  

In the end, this also enabled us to identify UX/UI issues, such as   ambiguous or misleading display elements for users, or certain fields in a form that would have deserved more robustness and protection. Thus, some of the bugs reported were not in themselves functional bugs, but raised issues related to the user perception.

Ethics in the remuneration of testers

An important point to stress is that, unlike other Crowdtesting platforms, testers are not paid according to the number of bugs identified. Indeed, this approach has 2 significant drawbacks from our point of view: on the one hand, if the tested application contains no or very few bugs to report, the tester will have spent time working but will not benefit from it. At the end of a certain period of time, when a bug could have been found by continuing the exploration, the tester may have stopped his work, telling himself that he is wasting his time. On the other hand, this may lead to reports of unqualified anomalies, because of the lure of gain that drives the tester to go back up.  

On the other hand, Testeum offers pricing based on the number of test cases and test environments (free for testers, targeted to specific OS/Navigators, …). Thus, at the time of ordering, the pricing conditions are as follows clear and without any unpleasant surprises. Testers can concentrate serenely on exploring the application and conquer the bugs!

Conclusion

Thanks to Testeum we had our first experience with Crowdtesting. This allowed us to discover the added value of this approach, mainly on the diversity of bugs encountered brought by the expertise of the testers. The tool is very intuitive, easy to use, and the analysis reports are very complete and detailed. Calling on a friend or relative to test an application is good, but calling on impartial and motivated testers will bring you more complete results! Testeum is a new and very promising solution, very intuitive and easy to use.  

Next week we will stay with HighTest, who will answer some questions about Testeum. Feel free to ask for comments in the meantime!