Frank Compagner, Senior Programmer over at Guerrilla Games has a new blog up. It looks like approx. 150 people are working on Killzone 2 right now and Frank blogs about how sometimes, the everyday hardware that he works with is being sabotaged by gremlins.
Hi there. My name is Frank. What's yours? Er, sorry, I still need to get the hang of this blog thing. I'll be the resident programmer on our squad, and as such I'm going to be pretty technical from time to time. Because I like getting technical. A lot. But not to worry, I'll try to keep things interesting for anybody curious about what it takes to get a game like Killzone 2 out the door.
Now, being paid to work on games definitely isn't the Walhalla you might think it is, but I'll tell you one thing: it ain't boring. Working on a game as big as Killzone 2 we get to have problems that few people have even thought about. All new and shiny problems, just waiting to be unwrapped so they can jump on us and make our lives miserable. One of the biggest problems we have is dealing with the sheer amount of data that's going into the game. There's about 150 people working very hard on the game right now and they're producing an ever increasing amount of high detailed models, animations, textures, audio samples and what not. All of this data goes into the central database (we use Perforce, which is a great tool, but does have its own peculiarities).
This all mostly works fine, but whenever somebody decides to do a "GetLatest" on his working files, he gets the combined effort of the other 149 people on the team copied to his computer. This can easily mean several Gigabytes of new data, which takes a considerable amount of time to download. During the download, the files on your hard disk are unlikely to be in a consistent state, so you basically have to stop working for the duration. This sucks, and we need to find a way to speed up the process, which is what I've been trying to do for the last few weeks.
We're already planning to replace our Windows32/NTFS server with a Linux64/XFS one. The new hardware has just arrived (I find myself strangely excited when looking at the rack mounted, all-black box filled with 15k rpm drives, should I seek help?), and now we need to configure it and find out how we're going to get the whole 3.5TB of our database safely onto the new server. And because few people have dealt with stuff like this, there isn't much advice to go on. This means lots of testing to make sure the server is running as fast as it can.
So I'm right in the middle of yet another test to determine the best configuration for our new Perforce server, when I get a call from my twelve year old son. He's in a panic, and I'm starting to get a little concerned, when I manage to calm him down enough to find out that the crisis is not, in fact, as life threatening as he initially made it out to be. He's recently picked up a nasty Civ IV addiction (I wonder where's he got that from), and he's just been invaded by his seemingly trustworthy neighbors, the Mongols. So now he needs my advice on how best to counter the threat. I do my best, but let me tell you: playing a game of Civ over the phone isn't the way that Sid intended. So I have to leave him fending for himself, while I finish the tests to determine the optimal size of the stripe set for the new RAID array that will hold our Perforce database (8KB seems to do the job best, but it's a close call). By the time I get home that evening his situation has deteriorated markedly, but after dinner we sit down together, and in an hour or so manage to recapture the lost ground. In an overly optimistic mood I promise him he won't have to go to bed until we crush the Hun. To his dismay, this decision is overturned another hour later by his mother when we get bogged down in a protracted fight for the Mongol capital. Still, with some more words of advice from me, he's able to finish the job himself the next day, and continues to score a space race win. Next difficulty level, please.
But back to Perforce performance. It turns out we have another gremlin lurking in our system: client-side disk fragmentation. The way that Perforce writes new files to disk under NTFS results in lots of disk fragmentation. Especially when dealing with large files (check) on disks with little free space (double check). We're working with Perforce to improve this, but that will still take a while. In the meantime we need to figure out how to improve the situation now. Really, it never gets boring when you're as obsessed with computers and performance as much as I am.
Next time: The joys of assert (woo, bet you're excited now!)
We would of gave you a short version, but it seems everytime you access the site you have to go through the whole age verification process and what not and find it yourself. But now you don't have to.
[Via Killzone]
0 comments:
Post a Comment