ISV Kaizen

Recently, we’ve published Dr.Explain v.3.0 beta for open beta testing. This is a major version update and we added a lot of really cool but complicated (for programming, not for users) features. So, we have to run comprehensive tests before releasing this update officially.

In this post I’d like to tell briefly how we run the testing process for the new version. Currently, we have not an army of QA specialists and dedicated testers so we use our team internal resources and the kind help of external beta testers.

Building a database of beta candidates

Every time I receive an e-mail with a support request, issue report or even uninstall feedback I wonder if this person could help us with testing of further versions. I use the following criterion for including him in our beta tester database:

• A user directly mentions that he wants to participate in beta testing of new versions
• A user actively suggests new features and improvements
• A user asks for a feature which is under development in the upcoming release
• A user has a rare hardware\OS\localization environment

If the user meets one of this criterion then I include him in the database. I try to keep the database up to date and relatively short - about 50 beta testers. I use Group Mail for managing the database and for bulk sending e-mails to testers.

Alpha version testing

Once the alpha version of the program is ready we send it to several most active users (addicts and evangelists) personally. At the alpha stage the program is usually quite rough and has bugs or “foo code”. Those selected testers tell if the new features are useful and well designed from their point of view. After this step we still can significantly modify some functions and parts of the program.

Beta candidate in-house testing

Once the beta candidate version is ready we spend several days for active testing inside the company. The best practice is involving people from other project teams. They have a fresh view and usually find lots of issues that were overlooked by developers. At this stage almost no new serious functionality is added into the version.

External close beta testing

After the internal testing and fixing, we build a first beta version for external testing. We upload it into the password protected section of the web site and create there a page with release notes and link to the file. That’s the time to use our beta candidate database. We send out invitations to participate in close testing of the product. We provide testers with login\password to enter the protected section of the site and to download the beta version of the product.

On the beta testing page of the website we write the following information:

• “As is” and “limitation of responsibility” disclaimers
• “What’s new in the new version”
• Contact information for sending test reports
• Download link to the latest build
• Release notes (changes log) for every beta build that we have published

Now we can update our tester database as well. After the bulk e-mailing, I remove records with bounced e-mails and with opt out responses.

Since that moment we start gathering feedback form beta testers. We upload new builds with fixes every a couple of days. About once a week I send invitations to beta testers to re-download the latest beta build. During the whole testing period I analyze web logs. Because all testers have unique logins I can see what latest build was downloaded by a certain tester. I record this information in the beta tester database as well. In the end of the beta testing I remove the records of testers who has never downloaded a file or, for example, has downloaded it only once and sent no feedback.

During the the close beta testing almost no new serious functionality is added into the version, only fixing and polishing.

Open beta testing

After we’ve almost stopped receiving the bug reports (the flow of feature requests will last forever ) we create the final beta build and publish it on our web site along with current version release. On this step, it’s possible to announce the availability of beta version among other users, on appropriate forums and blogs to attract as many people to beta testing process as possible.

It’s better to make a special section devoted to beta version on your web site to publish new build announcements, release notes and testing progress status.

During the open beta testing no significant changes should be done and all changes must be done very, very carefully.

Once you feel that everything seems OK with a new build - publish it as official release and start developing the next version.

Another interesting article on beta testing process:
How To Run A Beta Test… Or Not?

BTW, I invite you to participate in our public beta testing and to try the new Dr.Explain v.3.0 beta

Trackback URI | Comments RSS