ISV Kaizen

This is a short and simple plan for quick testing final release of your software application before making it public. Just not to overlook something trivial but very important.

Texts

Spelling: Check all text labels, menu items, hints and messages for typos.

Version: If you don’t increment your version or build number automatically during compilation then don’t forget to change it manually before final compilation.

Copyright: Check if the copyright year is the same as the current year.

User Interface (GUI) appearance

Large fonts: Test your application with Large Fonts (120 DPI) set in Windows display settings. Are all controls properly aligned and visible?

Different screen resolutions: Try your software under different screen resolution. Start with 800×600.

Positions: Check if all controls on your forms are positioned properly.

Icons: Test your application icon in all common sizes and color depths: 16×16, 32×32, … 16 colors, high colors, …. Does it look clear and smooth?

Basic operations

Keyboard support: Many people prefer keyboard to mouse. Is your application keyboard-friendly?

Tab order: Recheck the tab order of your controls. You may break the right sequence during GUI redesign.

Hot-keys: Are all common hot-keys supported?

Hot-keys activity: If some operations in menu or toolbar are disabled, are the corresponding hot-keys disabled also?

Double clicks in lists: In some forms, if you ask users to select a list item then check if double click works like single click+OK.

Data entry and storage

Editable drop downs: If you ask users to select a predefined value from a drop down list (combo box) and you allow only options from that list then you must check if the drop down’s edit box is read only.

Spin box and sliders limits and steps: Recheck all spin controls and sliders for valid minimum, maximum, and increment step values.

Spin box arrows: Verify if Up arrow button in spin box increases the value and Down arrow button decreases it, not vice versa.

Data type mismatch control: Check what happens if you enter unexpected values in data input fields, for example, a character string in a numeric field or a negative number in a field for positive numbers only.

Long strings: Enter very long strings in data fields and test how your software will handle it.

Empty strings: What will happen if user leaves some important fields unfilled? Will your program continue working correctly?

Clipboard operations: Does pasting objects of different media types from clipboard work properly in all fields? What happens if an image or rich formatted text is pasted in edit field?

Weird characters: Test if your software works and data remains valid if there are unexpected characters in input strings or files, e.g. new line character, tab, diacritic symbol, copyright sign, non printable characters, etc…?

Unicode and multi-byte languages support: Will non-English people be able to enter data in national languages and encoding: Arabic, Hebrew, Hieroglyphs, Cyrillic, Greek, …?

Local formats: Check if your software works correctly if user has different date, time, currency or numeric format: e.g. dd-MMM-YYYY instead of mm/dd/yy, or nn,nnn.nn instead of nnnnn.nn.

Debug options

Debug mode: If you develop in MS Visual Studio, for example, then don’t forget to turn off the Debug mode before final compilation and test your application built in Release mode. Some bugs, like memory buffers overrun, may appear only in Release mode.

Debug logs and dumps: Don’t forget to turn off all debug log or trace files.

Debug messages and alerts: The same here. Turn off all debug message boxes and alerts.

Foo data: Change all “foo”, “John Smith”, “Preved medved”, or “Loren ipsum” data to blank or actual values.

If you have ideas about other important points to check then post them as comments please.

Recently, we’ve published Dr.Explain v.3.0 beta for open beta testing. This is a major version update and we added a lot of really cool but complicated (for programming, not for users) features. So, we have to run comprehensive tests before releasing this update officially.

In this post I’d like to tell briefly how we run the testing process for the new version. Currently, we have not an army of QA specialists and dedicated testers so we use our team internal resources and the kind help of external beta testers.

Building a database of beta candidates

Every time I receive an e-mail with a support request, issue report or even uninstall feedback I wonder if this person could help us with testing of further versions. I use the following criterion for including him in our beta tester database:

• A user directly mentions that he wants to participate in beta testing of new versions
• A user actively suggests new features and improvements
• A user asks for a feature which is under development in the upcoming release
• A user has a rare hardware\OS\localization environment

If the user meets one of this criterion then I include him in the database. I try to keep the database up to date and relatively short - about 50 beta testers. I use Group Mail for managing the database and for bulk sending e-mails to testers.

Alpha version testing

Once the alpha version of the program is ready we send it to several most active users (addicts and evangelists) personally. At the alpha stage the program is usually quite rough and has bugs or “foo code”. Those selected testers tell if the new features are useful and well designed from their point of view. After this step we still can significantly modify some functions and parts of the program.

Beta candidate in-house testing

Once the beta candidate version is ready we spend several days for active testing inside the company. The best practice is involving people from other project teams. They have a fresh view and usually find lots of issues that were overlooked by developers. At this stage almost no new serious functionality is added into the version.

External close beta testing

After the internal testing and fixing, we build a first beta version for external testing. We upload it into the password protected section of the web site and create there a page with release notes and link to the file. That’s the time to use our beta candidate database. We send out invitations to participate in close testing of the product. We provide testers with login\password to enter the protected section of the site and to download the beta version of the product.

On the beta testing page of the website we write the following information:

• “As is” and “limitation of responsibility” disclaimers
• “What’s new in the new version”
• Contact information for sending test reports
• Download link to the latest build
• Release notes (changes log) for every beta build that we have published

Now we can update our tester database as well. After the bulk e-mailing, I remove records with bounced e-mails and with opt out responses.

Since that moment we start gathering feedback form beta testers. We upload new builds with fixes every a couple of days. About once a week I send invitations to beta testers to re-download the latest beta build. During the whole testing period I analyze web logs. Because all testers have unique logins I can see what latest build was downloaded by a certain tester. I record this information in the beta tester database as well. In the end of the beta testing I remove the records of testers who has never downloaded a file or, for example, has downloaded it only once and sent no feedback.

During the the close beta testing almost no new serious functionality is added into the version, only fixing and polishing.

Open beta testing

After we’ve almost stopped receiving the bug reports (the flow of feature requests will last forever ) we create the final beta build and publish it on our web site along with current version release. On this step, it’s possible to announce the availability of beta version among other users, on appropriate forums and blogs to attract as many people to beta testing process as possible.

It’s better to make a special section devoted to beta version on your web site to publish new build announcements, release notes and testing progress status.

During the open beta testing no significant changes should be done and all changes must be done very, very carefully.

Once you feel that everything seems OK with a new build - publish it as official release and start developing the next version.

Another interesting article on beta testing process:
How To Run A Beta Test… Or Not?

BTW, I invite you to participate in our public beta testing and to try the new Dr.Explain v.3.0 beta

There are problems in your software!

Otherwise your “demo installations / orders” ratio must be 100%, is it? If it’s 100% don’t read this post and write your own one about how you achieved this.

People are lazy and they will hardly write you by e-mail about problems they have with your program. They likely will remove your software from computer and will start evaluate a competitor’s product. The only chance to receive feedback from such users is to grab their attention during uninstall process.

There are many posts and discussions about how to properly implement the uninstall feedback. I’d like to write about our own experience. This is how the uninstall feedback is made in Dr.Explain - a rapid help authoring software.

Our setup program is made with Inno Setup.

We simply added the following code in the [code] section of .iss file:

[Code]
…
procedure DeinitializeUninstall();
var
ErrorCode: Integer;
begin
if MsgBox(’Why are you removing ProductName software? ‘ #13#13 ‘Would you like to answer, please?’, mbConfirmation, MB_YESNO) = idYes then
ShellExec(’open’, ‘http://www.yourdomain.com/unistallform.php’, ”, ”, SW_SHOWNORMAL, ewNoWait, ErrorCode);
end;

As you see, this hook opens a web form on the product website in browser. Many users don’t like when applications run browsers without their confirmation. So, we have to ask first if a user wants to open the uninstall feedback form.

This simple trick helped us gather valuable feedback from people and even to close several sales with those initially “unhappy” users.

Below there is a couple of related links about other ways of implementing uninstall feedback in Inno Setup:
More proactive way to force the user to write the feedback
Uninstall Survey plug-in for Inno Setup

Kaizen is all about continuous improvement. I’d like to acquaint you with a tactics that I use to find out what else may be improved in our Dr.Explain software and other our products. Besides feedback from users, which is very important of course, I use a trick that I call ‘A Saturday Newbie’. The tactics is rather simple but very effective.

Once a week, usually on Saturday, I remove our software, Dr.Explain, from my computer completely, i.e. program files, registry entries, and installed files. Then I go to the product’s website and from this moment I try to act as a surfer who has absolutely no idea what is the Dr.Explain software. I read the software description on the web site, explore samples, online help, and use cases, read reviews and testimonies. I ask myself: “Do I need this?”, “Will it work on my PC or Laptop under my OS?”, “How much does it cost?”, and other important questions. I look for the answers on the site. If I fail to easily find the answers on the website then I tell myself: “Dennis, this must be fixed asap!”

After the initial acquaintance with the product overview on the website I proceed to the download section (if I’ve decided that I need this product ) and try to download and install the demo version. So, I check if the download link is easy to find and if it works.

Now, the most interesting part begins. After the installation, which should be completed smoothly, I run the application and try to figure out how this pile of buttons, menus and controls can make my life better.

What should be my first step? Which button to press first? How to start a new project? Is there a sample? I ask myself a dozen of questions. If the answers aren’t obvious then I say: “Hmm, guys, you must make this more clear if you want my money.” Seriously, I try to forget that I’m a developer of this software and try to find a hint for my first action. Is there such hints in your product? This may be a wizard’s window, a pop-up ‘Click me!’ balloon, or even a simple ‘Start’ button (You know what I mean).
The same rule must work for all my further actions. After each step in the program I look for another clue: “What’s next?”. If I have no idea what to do next after a certain step I make a record in our To-Do list to make the transition from ‘Step M’ to ‘Step N’ more intuitive and transparent.

The main purpose of ‘A Saturday Newbie’ tactics is not finding bugs, but finding barriers that get in the way of a user when he tries to solve a certain problem with our software. Every time a user launches our software, Dr.Explain, he has a different purpose. Same here. Every time I act differently. Sometimes I act as if I need to create a new help file from scratch. Sometimes - as if I want to open an existing project made in the previous version of the program and to export it into CHM or HTML. Sometimes - as if I need to merge several existing projects. I play as many parts as I may realize. BTW, analyzing user feedback helps find out how people use the program.

Just a simple example, the Dr.Explain requires the Microsoft HTML Help Workshop to be installed on user computer for compiling CHM files. In the early versions we forced users to manually specify the path to the Microsoft HTML Help Workshop folder before enabling CHM file creation. That was a true barrier. Currently, Dr.Explain automatically detects the path to the Microsoft HTML Help Workshop on user computer and doesn’t bother users with this question anymore. The barrier was eliminated. During every testing session I find more new barriers that none of the developers or testers has seen before. Forget everything that you know about your application. Forget that you are a developer of this software and, I’m sure, you will also find many barriers in your program.

To reach the maximum effect you must not only play different parts but play them on different stages as well. Change your environment for every testing session. Use different OS versions, screen resolutions, hardware environments, and user privileges. Run your software both in registered and unregistered modes, and with all possible licenses (Free, Regular, Professional, etc.). Each environment modification may cause new barriers for your software users. You must find those barriers and eliminate them to make your software really easy to use and intuitive. I hope that ‘A Saturday Newbie’ approach will help you to do this.