Running a successful beta program

March 15, 2011

Yes, there are many types of software betas. In fact, Saeed Khan claims that delivering a successful beta program is one of the hardest things for a product manager to do. Why? Because so much is out of the hands of the PM and dependent on the engagement of the beta testers. So he has created a very thorough document on beta planning to help bring as much predictability as possible: Building a Better Beta.

Some key findings:

  • Recruiting is the most critical aspect of the beta program – get the wrong sites, wrong mix, not enough, and you are unlikely to meet the program goals
  • Usage: expect only 50 per cent of your beta sites will actually install and use the tool, and only 25 per cent will provide meaningful feedback (Joel Spolsky estimates 20%).
  • So that your sites aren’t just ‘playing around’ with the tool, provide them with example beta test scenarios based on expected product usage.
  • Evaluate beta product readiness by tracking usability, performance, installation, and upgrade path separately with their own criteria and requirements; simply tracking all open bugs and their severity leads to potentially arbitrary decisions about you are ready.

The paper also notes that betas need to involve the entire organization. Dave Daniels identifies that this helps answer key questions for beta program: Is the product positioning and messaging correct? Are your delivery and support teams ready?