Running a successful beta program

Yes, there are many types of software betas. In fact, Saeed Khan claims that delivering a successful beta program is one of the hardest things for a product manager to do. Why? Because so much is out of the hands of the PM and dependent on the engagement of the beta testers. So he has created a very thorough document on beta planning to help bring as much predictability as possible: Building a Better Beta.

Some key findings:

  • Recruiting is the most critical aspect of the beta program – get the wrong sites, wrong mix, not enough, and you are unlikely to meet the program goals
  • Usage: expect only 50 per cent of your beta sites will actually install and use the tool, and only 25 per cent will provide meaningful feedback (Joel Spolsky estimates 20%).
  • So that your sites aren’t just ‘playing around’ with the tool, provide them with example beta test scenarios based on expected product usage.
  • Evaluate beta product readiness by tracking usability, performance, installation, and upgrade path separately with their own criteria and requirements; simply tracking all open bugs and their severity leads to potentially arbitrary decisions about you are ready.

The paper also notes that betas need to involve the entire organization. Dave Daniels identifies that this helps answer key questions for beta program: Is the product positioning and messaging correct? Are your delivery and support teams ready?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: