Articles & Editorials: Beta Testing & The Perfect Product

By now you have probably read GenKiller’s article (what?? You HAVEN’T read it? Shame on you… Go here to check it out), this article talks about how (in my opinion) Westwood could release that great new game that GenKiller says Westwood needs.

How? By using Beta testers. Not your normal in-house Beta testing but external Beta testers from the community.

Please understand that even though I may be a little critical of the fact that Westwood doesn’t use external Beta testers I have the utmost respect for Westwood as a company – and a little criticism can be constructive.

Today, with everyone rushing to release their product as fast as possible the end users (us) will often find themselves left with software full of bugs. Some companies are aware of the bugs before they release so they tag the software as Alpha or Beta staged software. Other companies just never leave Beta (see: icq) though that can be for a number of different reasons (liability, selling the end product etc.). In the gaming industry releasing a Beta product to the public is sometimes problematic. Whereas some companies will actually make you pay for Beta software (usually with cheaper upgrade prices to the full version) this attitude will not work with a gamer – who, especially in today’s Internet Warez world, is only going to spend money on a final product.

A gaming company can’t afford to release a buggy game either. Relying on players to find bugs after the game’s release and then fixing them with a stream of never ending patches is not the ideal way to do things – this can (and will, usually) wreck the company’s prestige and the basic faith that gamers have in the company.

I read an article somewhere on the net by someone saying that we should all buy games that are known to be good – but the games that are just not up to standard, we should all be able to copy them. Some philosophy, there are just to many games out there that are not worth a gamers money (not that I am saying that we should all go and flood warez sites, I just say that if the game is bad it isn’t even worth the download time).

Back to beta testing… commonly known as QA (Quality Assurance) most companies have in house Beta testers who are supposed to use the product and also test specific areas of the product. The place I used to work at had one such place, that were so good that the bugs list was always full (and was always being fixed which is also good). Thing is, there were only three people and even though the program looked relatively simple it had lots of background features that made the program very bug prone.

What was the solution? We needed to find the most bugs, in the most test situations without hiring a Microsoft sized in-house Beta testing team. We released a private Beta. A lot of the people who got the beta didn’t help much – though in the end it was worth it when we got bugs like “While the program is loaded Solitaire won’t run”. None of the team (or us, the programmers) thought of loading Solitaire while the program was running (btw. the bug was a MS one 🙂 ).

Taking Microsoft as an example… They have a giant QA team, but they still get thousands of volunteers to beta test their software. I recently participated in the closed Beta for Windows Millennium Edition – and from the number of bugs (that weren’t necessarily hardware specific) that I saw people posting on the numerous newsgroups you notice that no matter how big the in-house team is, an external beta tester who may be your every day Joe Dude beta tester (volunteer of course) might (and probably will) still find something that the in-house team missed.

So, why am I telling you all of this? I think that maybe Westwood should learn a bit from Microsoft (yeah, yeah, no MS jokes 🙂 ) and think about an external beta testing team. I am sure that the in-house team is one of the best you can get but they can never do everything – the obvious example: Tiberian Sun – the team didn’t notice the fact that the Nod Artillery was a little to strong and didn’t really miss? Or that the the two sides weren’t really balanced?

I think that Westwood should open the doors to a closed external Beta test. To make sure that the Beta doesn’t leak to the outside each copy can have some ID that will identify it. To be even more careful Westwood should only release the Beta to people who have been in the WW Fan Site community for a long time, everyone from Webmasters, News Updaters to people who are seen a lot on Forums around the community. I doubt that anyone in the community will want to release a Beta that he personally got from Westwood and I also doubt that he would risk doing so if Westwood could automatically say who’s copy was leaked out.

I also think that if Westwood doesn’t release a Beta then it should at least explain it’s philosophy (regarding external Betas). Quite a few companies lately have release either close or public Betas of their software. Blizzard had two closed Betas (one of the game and one of the via. the Net module), ID release a public Beta of their game Quake 3 and <whatsisname?> also released a public Beta of Unreal Tournament (admittedly these were Demos but they asked for bugs).

From where I stand, an external Beta is the way for Westwood to go to release the next hit. Agree? Disagree? I want to know about it! Send me an e-mail or go to to the Forums and post your opinion.

Articles & Editorials Index