Lessons learnt from beta testing

Last week, saw the completion of the first round of beta testing for Bean Counter. Was the beta testing a success? No, not really.

What went wrong?

I made a few mistakes in terms of the testing.

• I accepted anyone into the testing programme
• I didn’t create a set test plan for them to follow
• I didn’t provide any clear feedback procedure

The biggest error of judgement was simply to accept anyone who wanted to test the application. In reality, it only makes sense to test people who will either test the application really strongly and try to break it or people who form part of your target market.

I chose people who, despite their enthusiasm, didn’t actually do very much testing. Sorry guys, if you’re reading this but here are the stats:

• 10 beta testers
• 5 didn’t even look at their test page
• 4 people installed the system but didn’t get any further
• 1 person installed the system went away and logged back in the following day

These stats left a problem – what were the reasons behind their lack of interest once being given the keys? It could’ve have been a number of reasons: a lack of time, a confusing application, a lack of a test plan. At this stage I do not know why.

I was so excited that anyone wanted to beta test my application and give up their time to help me for nothing. Beta testing does seem to create a buzz and if someone asked me to beta test a new web app that looked remotely interesting – I’d probably agree too – but I’d probably also lose interest when it quickly became obvious that I’d have to put some effort in for literally no reward.

No test plan

I suspect a missing test plan didn’t help. I foolishly asked people to simply play around with the system and casually email me back with their thoughts. This isn’t a good way to do things – without some kind of formal procedure, people are unlikely to test an application properly add to the fact that these testers were vetted – I have no idea if they wanted from the system what the system was offering.

No feedback procedure

By failing to create a formal procedure for feedback and relying upon casual email correspondence, meant I got very little feedback from the testers. Nobody emailed except maybe to say it was good or ok. Which when you’re trying to find bugs and see how useful something is – isn’t very helpful.

What I should have done

Next time I beta test anything, I’ll be choosing the testers very carefully and finding some way to reward them properly. I’ll be creating a test plan and asking the testers to make sure they perform a few key tasks and finally I’ll be writing some forms (preferably built into the application) that allow for quick and easy feedback possibly in a questionnaire format with simple options.

The (almost complete) failure of this test was my fault and not the testers’ and I don’t want to appear as though I am blaming other people for the mistakes I made.

There was one saving grace from the trial, I did learn that my installation technique for multiple customers needed improving, which it now has. At the start of the trials, setting up a new account took about 20 minutes whereas now it takes 5 minutes.