June 27, 2005

Useful Middleware

Welcome, Rob, and congratulations on joining that small simulation studio!

I have a lot of stuff I want to post about the AIIDE, but since you've got me on the topic, I want to talk about automated (and semi-automated) content generation and its new poster child.

Will Wright's presentation was clearly the highlight of the show. Seeing Spore and hearing Will's thoughts on the AI/game design interface and new approaches to content generation was like basking in a warm golden spray of game development wisdom.

I've heard some allegations that Spore is unlikely to the reach the mass-market appeal of The Sims (though really, what could?), or that the user-created content will be rife with mediocrity.

As far as I'm concerned, though, Spore has already proven its worth just by pushing so many game developers to think differently about game design and development.

I am thinking a lot recently about the way Spore deals with animations. You can apparently design pretty much any kind of creature you like, and the engine will automatically figure out how to animate it. In order to accomplish this, the game is using -- (and if anyone has more information, please correct me if I'm wrong on this point!) -- a highly advanced form of fairy magic.

This is something that I find very exciting, as the magic of the fairies (and their close relatives, the leprechauns) has long eluded me, despite many years of relentless searching.

At the very least, their engineers clearly are in possession of testicles the size of large grapefruits. Not many engineers I know would sign on for a project that required them to construct novel animations at runtime for creatures of arbitrary shape and complexity.


And this leads me to thinking about middleware.

We have a ton of creatures in the Metroid Prime series. Prime 2 has something on the order of 40 unique creature types.

So anything that could help us create more creatures faster without sacrificing quality is exciting.

AI is highly constrained by the animations: I have these 30 animations, and my creature's entire range of expression has to come from those 30 animations (obviously sound and particle effects are a big part of the picture too, but those are also usually limited by the animations at hand). When you only have a small number Lego pieces, you can only put them together in so many ways.

Procedural animation is one thing, but it's outside the scope of this post, and don't think it's feasible in the near term to consider using procedural animations for most of what animators do by hand.

Right now I'm thinking about kinds of automated content generation specifically for animations. If there's a way we can help our animators create more animations in less time without sacrificing quality, I want to know about it.


Most AI "middleware" doesn't seem to me to be terribly useful. A lot of basic pathfinding stuff, flocking and crowd simulation, HFSM (hierarchical finite-state machine) editors, and scripting systems. Mostly things developers already have or can develop and customize better themselves. Most AI middleware grabs the "low-hanging fruit" of game AI development, and you still have to do the hard stuff yourself.

AI for any game is built on top of some underlying engine technology, and is built in the context of some a game design. If either one of those things changes significantly, it can break your AI completely.

On the tech side, if you change the physics simulation, the world representation, the game object representation, or the animation system significantly, you may very well have to toss major parts of the AI, if not rewrite it from scratch.

Similarly, on the design side, a huge amount of the work in AI involves tuning and tweaking it for your specific game. The real challenge in game AI is customizing it to be a perfect fit for your game's design. And even seemingly minor changes to the game design can have huge ramifications on the AI. "Oh, yeah, there's a new jetpack that lets the player fly up to 200 meters, and the AIs have to be able to chase him through the Ice Caverns. Didn't you get the memo?"


I cornered Will at AIIDE about the animation-generation system in Spore. There's really no published research at this point -- it's an iterative process between the engineers and the animators.

Engineer: "So, um, how would you animate this 5-legged creature?"

Animator: "Well, you'd move this leg first, and turn the foot like this ..."

Engineer: "Right, OK ..." (three hours later:) "Sorta like this?"

Animator: "Ummm, yeah, kinda like that, except it raises leg #3 a half-second too soon."

Engineer: "Got it. ... Oh, by the way, you don't mind that I'm automating you out of a job, do you?"


I'm not referring to procedural animations so much as tools to create significant quantities of animation content automatically.

I don't think an "artist in a box" is feasible or even desirable. And when you're working with animators as good as Retro's animation team, you want to ensure they have full control over every last bit of animation content that gets into the game.

But I do think there have to be some great ways to augment the work the animators do, and give them some power tools that will let them sometimes work at a higher level of abstraction when it's appropriate to do so.

I'm aware of some work by Demetri Terzopoulos and others on automated physics-based locomotion and learning, and some work by Wu and Popovic on Realistic Modeling of Bird Flight Animations.

If you've played World of WarCraft, you've probably taken a flight path from one town to another at some point. You ride a giant bat or griffon on a fixed spline path to the destination, and the creature flaps and glides as it follows the spline.

The problem is that the creatures have a noticeable tendency to animate incorrectly -- they often use the "glide" animation while flying up a steep hill, and the "heavy flapping" animation while gliding or coasting downwards (of course, it's a testament to WoW's exceedingly high production values that I'm even able to nit-pick about a detail like this, which a lot of players probably don't even notice).

Memory and performance concerns aside, it's not hard to imagine applying Wu and Popovic's work to flight paths like this. I'd like to see a system that could take as input a tagged skeletal model of a giant bat, a bunch of tunable physical parameters, and a spline path in the shape of the actual spline path of the flight from Orgrimmar to Gadgetzan, and generate one huge animation that's completely customized for that flight, and could be streamed from disk in-flight.

Of course, locomotion is only a small piece of the animation puzzle. A lot of the most important animations are emotive, and contain a lot of expression that's a direct result of the animator's creative intentions rather than anything involving physics. Real-world animals spend an inordinate amount of energy on things that seem amazingly counterproductive if you only view them in terms of minimizing energy expenditure or maximizing survival selection -- but there are very good reasons for them if you can think about them in broader terms.

At that point, the problem shifts from the difficult-but-not-intractable problem of simulating the physical interaction of a creature and its environment to the neurological-exception-throwing problem of trying to simulate the net result of five billion years of evolution.

So that limits the usefulness of methods based on preprocessed physical simulation. I'm curious to see whether Spore's automatically-generated animations have as much personality as the creatures themselves, and whether there are specific animations for emotive taunts, mating dances, and other fitness displays.


So that's what I'd like to see: AI middleware that can take skeletal models as input, combined with a bunch of tunable physics parameters and possibly a mesh for an environment to describe a fixed physical constraint, and produce animations based on some combination of physical simulation (as with Wu and Popovic), motor control system evolution (as with Terzopoulos), and/or expert systems based on animators' expertise combined with highly advanced fairy magic (as with Spore).

Clearly I'm talking about several different systems. Bipedal creatures animate differently from quadrupeds, and those animate differently from creatures that that slide and slither, which animate differently from creatures that swim, fly, or swing from tree to tree. So the interaction with the physical simulation is different in each case and requires totally different sets of parameters. Any one of these is a huge research project in its own right.

In the case of Wu and Popovic's work, it's not hard to imagine a kind of middleware (Maya plugin, maybe) that could take a skeletal model of a flying creature as input, allow me to mark it to identify the different body parts (wings, body, tail, etc.), specify the mass of the various parts of its body and other physical parameters, and feed it a spline path that the flying creature needs to follow. It would run the simulation as much as required to output a maybe-not-totally-physically-correct-but-highly-believable animation for the creature following that entire path.


It would also be great to see animation middleware that could create animations based on automated generation and user selection.

Imagine I'm an artist, and I create the skeletal mesh for a creature and a set of 10-20 standard animations for it -- walking, running, turning, dying, taunting, and so on. I also tag various joints with names like "head," "hand," "foot," "tail," etc., along with a lot of physical parameters for the general range of motion for the various joints.

Based on that, it's not too hard to envision a system that could then begin semi-randomly animating various parts of the creature's body, using the tags and existing animations as guidelines. Essentially a form of evolutionary art applied to animation. The system builds new animations by combining parts of existing, artist-built animations and performing random mutations and crossover on the animation dataset, and then repeatedly asking the user, "Which looks better, A or B?"

Essentially this is a kind of genetic algorithm, but using the animator's initial set of hand-crafted basic animations as the initial population (as well as a guide for what kinds of motion are most likely to be legal when performing the "mutation" operation), and then using the user's feedback as the fitness function.

The first 50-100 animations would look terrible, but eventually, you'd start to see some interesting stuff. Ideally, the process would be fast enough that it could let an animator generate animations more quickly than building them by hand -- or at the very least, come up with some novel ideas the animator might not have come up with himself.


But mostly, I'm hoping the Spore team will publish some info on what they're doing and how they're doing it.

Next year's AIIDE, maybe ...

Posted by PaulT at June 27, 2005 01:59 PM