Gears of War 3: Rod Fergusson's Beta Post-Mortem

Gears of War 3 Executive Producer Rod Fergusson talks about the virtues of beta-testing GoW3

Sounds like the Gears of War 3 beta was a tremendous success...
Yeah, it was an awesome success from our perspective. We were able to do a lot in terms of the testing – we had a little under 1.3 million players play online – and then to be able to take that type of scale and apply it to our game this early in development has allowed us to make a bunch of changes and tweaks around dedicated servers and matching-making, the kind of thing where you just can’t get that information when you’re testing with 100 people or 1,000 people. But when you bring a million people to bear and you have concurrent rates of 100,000 people playing all at the same time, then you get a sense of what you really need to do to optimise your use of dedicated servers, to make sure you’re matchmaking in the best way possible.

How did you even begin to go through all the data and stats you collated?

Oh yeah, we’ve been going through it from the day it started. We were able to collect data from the first day, and even in the week of private testing when it was just Microsoft and Epic before it went public for Bulletstorm preview week we had already made a bunch of changes. From day one we were looking at what are the things that we can change and what needs to be changed to tweak and tune as we go along.

“We really put a ton of effort to try and create this sticky experience that people want to keep play”

One of the big things we didn’t want to do was react too quickly to things like weapon balance, because people were kind of bringing their Gears 2 sensibilities to the game and kind of defaulting to original tactics and we knew that people had to learn there were other weapons, learn new strategies and tactics. If on day one we get a bunch of feedback about ‘ohmigod, rifles are so overpowered because all I use is a Gnasher and I’m still dying...’ it’s just because people are defaulting back to Gears 1, Gears 2 mentality. We had to let it sit and let people play it for a couple of weeks.

Was it tough to just leave people to it and hope for positive feedback?

Yeah it was. Obviously, we were a little bit nervous because of the issues we had when Gears 2 first launched so we were nervous as we rolled out each wave, when the Bulletstorm people came, then the pre-order people... at the end of the day it went pretty smoothly. There were a few times we had to remind people it’s called a beta test for a reason. We had a couple of days when we were doing maintenance on the dedicated servers and doing some tweaks, but as I was trying to explain, having the servers down for a day during the beta is saving us a bunch of time and effort ahead of September 20th when the game actually releases, so it was actually a good thing we found the issues we did rather than after release.

How would you respond to critics that have likened these large-scale betas to little more than a promotional tool?

The idea of a beta came from PC gaming – being able to release an unfinished product to get feedback on so you can make a better product has been around for a while. The mentality of console gaming is really not so much this idea that you can have an unfinished product on a console, so a lot of people look at betas now not as technical betas but as actual demos, and they’re going to make a purchase decision on how good a particular beta is. So that was one of the reasons why in Gears 1 and Gears 2 we weren’t able to do a beta, because you have to have such a level of polish now to do a beta that it’s pretty much near the end of the game when you’re at that level of polish, and at that point it’s too late to take feedback and change a bunch of stuff based on player involvement.

The extension we got from April to September gave us that window of opportunity where we were already polishing up for our April release so when we got the extension, we thought ‘well we’ve got this polished up experience and it’s really not bad. If we can get it out in players’ hands we can actually get feedback and respond to it before we get it out there in September’. So the planets aligned, and we were actually able to get out a very highly polished beta, for both purposes. If you look at over a million pre-orders, obviously the beta struck a chord from a demo perspective for people making purchases, but it also served a huge role for us in terms of testing dedicated servers, weapon balance, map layouts, all that kind of stuff.

Do you think that the wider use of console betas could stamp out glitchy, unfinished games?

Yeah, I believe so. It’s hard from a platform perspective. Understand you have that certification process there for a reason – so that anyone coming to your platform has the best experience possible and because you don’t want people to look negatively on an Xbox and say ‘oh this game was unfinished and I blame the brand not the individual developer’. So I get that issue, but I do see that when you take a game that’s on any console and on PC and you look at the different rates of updates, you can see they update much quicker on PC because of the fact that there isn’t that certification process.

If you were to find a bug in your game today, from the moment you spot it, to the moment it gets released can be up to three to four weeks just because you have to creating a title update, getting into certification, testing and everything else... It makes it harder to do those kind of unfinished products where you can respond overnight. That lack of a quick turnaround makes it tougher for sure. I think things would need to change from a platform perspective, but also a customer perspective, because I don't think a console player is as resilient as a PC player in the sense of... a PC player is used to swapping out video cards and installing new drivers and the DIY model, so getting an unfinished game - they're sort of used to that experience, where console gamers are much more like 'it's a closed box, it's all finished goods it needs to be perfect out of the gate'.

Can you tell us about any changes you've made as a result of the beta?

Yeah, We changed a bunch of things. We changed some weapon balance on the guns – if you got in on the last weekend you would have experienced a lot of that change in terms of how the guns felt relative to each other. We found Thrashball and Trenches had some issues with people being trapped in spawn locations, they were getting spawn-camped. So based on the feedback of players we adjusted the layout of the maps to allow for other passageways to get out so you weren't trapped in that one hallway anymore.

If you were to find a bug in your game today, from the moment you spot it, to the moment it gets released can be up to three to four weeks just because you have to creating a title update, getting into certification, testing and everything else... It makes it harder to do those kind of unfinished products where you can respond overnight. That lack of a quick turnaround makes it tougher for sure. I think things would need to change from a platform perspective, but also a customer perspective, because I don't think a console player is as resilient as a PC player in the sense of... a PC player is used to swapping out video cards and installing new drivers and the DIY model, so getting an unfinished game - they're sort of used to that experience, where console gamers are much more like 'it's a closed box, it's all finished goods it needs to be perfect out of the gate'.

We adjusted the time it takes to capture rings in King of the Hill to allow for tasters matches. The 170-point limit was just a little too long, so a lot of time matches went into overtime unnecessarily. A lot of the feedback we got revolved around things like how we value score; some people felt like completing the objectives in the game like capturing the leader or capturing the ring weren't being valued enough over kills and so people were still getting distracted by just playing team deathmatch inside of King of the Hill – they're not actually going after the objective. So we focused on that.

One of the first things we got feedback on were some of the ribbons we had – a couple were meant to be underdog ribbons, like ‘hey, congratulations, you killed a guy that ws 50 levels above you, or you killed a guy that was 25 levels above’. A lot of players are completionists, so a big piece of feedback we got right away was ‘if I join your game on day one and I play religiously then I’ll never get that ribbon, because there will never be anyone 50 levels above me.’ So we decided, ‘okay, we understand’ and we ended up removing those two ribbons and putting in different ones, for those with that completionist sense, so they can say they got every ribbon.

How did you go about listening to feedback during the beta, and how did you decide what was helpful?

It’s a mixture of listening to the forums, listening to Twitter and you have to interact with the fans, because there’s a lot of great stuff and a lot of times perception is reality. Also, one of the great things about having all the data we did in the backend is we’re able to say ‘well actually guys if you look at the data in terms of number of kills [the weapons] are pretty equal, or in terms of downs, or whatever… you use that perception issue to say ‘I think you have a problem’ and then you can dig in to see if the problem’s real or not, and then you can look and say ‘well actually it isn’t, it’s just their perception or their context makes it feel that way’.

Did any stats surprise you?

It was surprising to see how quickly people were going up in level. In such a short period of time for people to play so much… [laughs] that was one of the things that was kind of mind-blowing, the stats we put out recently that there was 249 years worth of matches, which was just like ‘wow, that’s crazy!’. For the most part the stuff was within what we were kind of expecting.

One of things that was really interesting was understanding the difference between a public match and a ranked match. A public match was really meant to be about quantity of play – you go in and you join in progress, you play, and you just keep going through the map cycles and you’re just in for the fun of it. And if you can’t find a match then you go in a play with bots for a little while until overtime when they’re replaced with people.

Ranked matches are about quality - you need to have two full teams, it match-makes the teams based on their TrueSkill level from Xbox Live. That’s a much higher quality of match, but you can’t play the same team twice because of concerns over cheating and all this stuff. So it’s this notion of quantity versus quality.

First we had the dedicated servers spilt 50/50 for either game type, but overwhelmingly people were playing public. So over the course of the beta we ended up making an 80/20 rule where 80 per cent of the servers were public and 20 per cent were ranked. It’s interesting because it’s the opposite of what Gears 2 is right now because ranked is the predominant way it’s played, where as it was public in the Gears 3 beta.

Could that change for launch?

It could change a little bit. I don’t know that we’d necessarily go out with that heavy skew, I think we’d be a little more balanced and account for the fact that people will want to try both sides. It makes sense to me, one of the things that we really tried to do with Gears 3 is make it more accessible and the idea of team death match and having this public mode, which is something we’re changing the name of right now in the game to ‘quick match’ because it’s just that idea of getting in and playing.

Are there any ways in which the beta informed the campaign?

Not really. The campaign aspects are quite separate from multiplayer. Even some of the weapon balance; the things you have to do on the sawed-off shotgun to make it balanced for multiplayer you can sort of undo in the campaign because you want it to be fun and you don’t care that the player has an advantage over the AI. So there wasn’t really a lot of crossover from what we learned in the beta.

But we learned a ton for multiplayer, and that’s what I’m really excited about – in the past we’ve talked about how the lion’s share of our effort has gone into the campaign and small amount of effort in the multiplayer relatively speaking. But this time around we really put a ton of effort to try and create this sticky experience that people want to keep playing with ribbons and medals and levels. Just making it more accessible has been a huge focus this time around.

Do you feel fortunate in being able to pull off such an operationally huge beta test?

Yeah, absolutely. It was really because of the schedule extension that we even had the ability to do this, and then to execute on it and learn all of the stuff we learned – it’s been really beneficial. It’ll make the game we’re going to ship on September 20th all that stronger. It’s a really big game; the biggest campaign we’ve ever done, arcade mode, beast and horde and competitive multiplayer… it’s packed full of goodness. It’s nice to be able to take that one thing that would have kept me awake on launch night wondering how the multiplayer would be… this is going to let me sleep.

Source: nowgamer.com

like it!! +1

good interview

They should've gone into more depth on changes instead of just talking random stats, and about the servers... :/

But, eh; pretty good.