How games have learned from art, music and maths – and how the world might learn from games
And on the third day, God made the world. And He saw that it was rubbish. And He then looked at the other worlds that He’d made, and thought, “They’re a bit bobbins, too, infallible as I am.”
So He pondered for silent indefinites, as the void drifted tetchily on, waiting for time, space, gravity, bicycles and all the other concomitants of virtue and vice. Eventually, He got a bit frustrated. “Bollocks to it all,” He thought, blasphemously. “Handcrafting a universe is for losers. I’ll procedurally generate an infinity of them and just choose the best one.” And lo, that was the morning of the third day, and He saw that it was good. So He went and watched fractal zooms on YouTube for the rest of the week.
As a child, I remember being fascinated by Mandelbrot. I didn’t at that time know what the word meant, except that in the corner of our BBC-riddled IT room at school was a single, fast Acorn computer that the IT teacher treasured and guarded. The only thing he seemed to run on it was a Mandelbrot simulation, which we used to sit and watch over our lunch break. It would take ages to draw each screen as it zoomed in on part of the structure, which would always reveal more structure and more complexity. Eventually, sometimes, it would repeat.
A Mandelbrot drawing is a representation of a mathematical set – the set of complex numbers that do not tend to infinity; which is to say, equations involving imaginary numbers; which is to say, numbers that involve the square root of -1. Which is to say… that I didn’t take maths past A-level and this is really difficult to understand. So I promise there won’t be any more equations in this article. Really.
More interestingly for our purposes is that, whatever else Mandelbrot is, it’s an equation that throws up astounding complexity and beauty without the user having to create anything themselves. Look closely and you can see it features spirals, valleys, seahorses, tentacles, antennae, double-hooks, islands. The mind’s eye sees cityscapes, ships, people holding hands, trees and organic structures that have never existed. And all this is created from an equation?
Well, only with a computer. Fractal art, like Mandelbrot, mostly exists as an outgrowth of using iterative methods to solve polynomial equations, and that really requires a computer. Looking at generative pieces, it’s hard not to agree that they’re art – but the artist has at most set the parameters for their creation, not handcrafted the piece, or in this case, hand solved the maths.
Similarly, ‘generative music’ can be constructed without direct user input. One could also argue that Bach’s fugues were generative, given that they were based on a strict underlying mathematical process out of the control of the creator. There’s even generative writing, which has grown from the ‘cut-up’ techniques of writers such as William Burroughs or Georges Perec, and France’s mechanical-writing Oulipo school.
That’s the heart of procedural generation. It’s grown out of maths, through art and music, and it’s taking over gaming. Where will it stop?
“People have always been fascinated with creating unpredictable things by following rules,” explains Michael Cook, an academic who specialises in procedural game design. “Even children’s games where you pass along bits of folded paper and each write part of a story. It’s all about being delighted at something strange coming from nowhere in particular.
“We’ve dreamed of machines that make things for a very long time – Ada Lovelace wondered about machines that could write music long before Alan Turing was even born. I think there’s just something magical about the idea of something coming from nothing – and I still feel that magic each time I work on a new generator!”
A ROGUE’S HEART
Cook’s main project, Angelina, is a special case, which we’ll talk about later. But he’s right about our love of the emergent outgrowth of rule structures. “Of course, there’s nothing better at following rules than a computer, so it didn’t take long for programmers to give computers recipes for generating complex, beautiful things. Computers let us transform theoretical patterns and ideas like fractals and L-systems into amazing machines with moving parts that could be poked and examined and executed again and again.”
The most standard usage of procedural techniques in modern gaming is in rogue likes and indie games, such as Minecraft, Dwarf Fortress and NetHack. It’s an area that Nicholas Vining’s Gaslamp Games specialises in, and he’s adamant that it’s essential for certain genres.
“For the games we make,” says Vining, “you have to have procedural generation. Both Clockwork Empires (a steampunk colony builder with a Lovecraftian twist) and Dungeons of Dredmor rely on having a lot of replayability, and a good way to get replayability into a game is to have it change every time. You really want to have random geography and random levels and random stuff being thrown at you.”
But why use procedural generation? Surely we’ve got by perfectly well handcrafting them so far? “People want giant experiences, and value for money,” says Vining. “AAA games deal with this by giving them 100 hours of hand-massaged content – Dragon Age: Inquisition is the most recent example – but that doesn’t scale unless you’re making, you know, Dragon Age. (And then you’ll end up probably having to make money by DLC.) Being indies, we have to cheat. And frankly, hand-massaging 100 hours of content is more laborious than it needs to be.”
ON THE TUBE
Vining is also keen to make the game easy to share and non-repetitive – not merely for the end user, but to attract the YouTubers who mostly drive a game’s fame today.
“We’re now making games for players who want to also show themselves playing the game, on YouTube, and doing stuff. I think you’re stuck with that nowadays. I think Minecraft did it to us. So, you have to make a game where YouTuber X playing the game can have a completely different experience than YouTuber Y.
“If you look at the stuff that gets put up on large YouTube Channels, it’s things with a creative element, a procedural element, and also Goat Simulator. Procedural content gives you just enough hooks upon which a good storyteller can hang their own hat. I think that’s something that Minecraft either got or stumbled upon – there’s a certain level of crudity needed where you can then have a child just stick their imagination on it, and have something wonderful happen, and a lot of the people gunning in the same space have never really figured that one out. “
LIMINAL PROCEEDINGS
By contrast, Dr Tom Betts’s motivations are less about speed or fame, and more about interesting outcomes. Dr Betts started as an arts student at Goldsmiths College in London. “I was really interested in the idea of creating systems that would make art work for me. I’d always been into computers and games so it made sense to combine those interests. I taught myself to program and I’ve been making generative and proc-gen software ever since.” Dr Betts has created art, performed music and has been the lead programmer at indie games company Big Robot since 2008.
“I find it interesting that working with proc-gen is always a sort of collaboration between the programmer, the program code and the player. You can’t dictate the outcome as precisely as you can in traditional production methods, and that makes it more interesting. Also, in terms of art and music, generative/proc-gen work has its own sort of aesthetic that comes from its ability to produce large numbers of variations on a theme, all of which are different, but maintain a shared style.”
Several of Dr Betts’s games have involved procedural generation. Sir, Are You Being Hunted, for example, uses a procedural system to generate all the terrain, housing, flora and fauna in its open world. “Procedural generation can aid the creation of all kinds of game world, especially in the case of large open-world games or anything with rogue like aspects. It can help smaller teams build larger environments and it can automate or assist many of the processes involved in larger studio production pipelines. We will undoubtedly see the usage of PCG tools and methods grow as games increase in complexity and production cost.”
INNER EXPLORATION
Cook has yet another motivation to use procedural generation – the reintroduction of genuine exploration to games. “Minecraft showed an entire generation of gamers how procedurally generated worlds could inspire a sense of discovery, exploration and awe – there are similar revolutions around the corner for generated game mechanics, generated stories, and so on. It’s incredibly exciting.” Part of that is there can be no walkthrough to a proc-gen world – at most, some recipes and best practice recommendations.
Cook is also interested in the development of aids to game development, which also come under the heading of procedural generation. “There’s a lot of interest in building software that can help people design games, either by learning how humans design, or using a big database of rules. For example, imagine feeding every Team Fortress 2 level into a machine-learning system, and then using that system to evaluate, comment on and even adjust levels designed by modders.
“The other big thrust is the automation of new areas of game design, moving out of the same old areas like dungeons and levels and looking into generating other, harder kinds of content. I’m really excited by this move – there are genres out there that we can’t imagine right now because we’re waiting for new generators to come along and show us the way. Last month at a conference I saw systems which can generate metaphors, cocktails, dancers, fables and jokes. I’d love to see all of these things built into games, games that can generate new kinds of content beyond trees and rocks. I want games to write theme songs formy MMO character. I want games to write myths about my epic journey. I want games to procedurally generate me a best friend I can talk to. I want to see new things coming out of our computers.”
BRIDGE OF SIZE
Vinings agrees, particularly when it comes to music. “Repetitive music drives me nuts; once you’ve listened to Dungeons of Dredmor’s entire soundtrack, good as it is, you’ve heard it all. We tried to modulate this in Clockwork Empires by having the soundtracks adjust themselves as your mood changed, and while it does work pretty well, it doesn’t work as well as it could if we iterated on the technology for a few more rounds.
“The problem is that you have to have somebody who is a musician and a programmer, and you have to build tools. It’s a very complex process that requires specific sorts of individuals. How do you teach a computer about orchestration?” Despite that, we have taught computers about things like image recognition (see “Dream of the Deep” below).
One of the most interesting procedural generation benefits has to do with file size. Today we’re growing accustomed to an endless increase in our computer’s storage capacity – 1TB drives are now commonplace. But in areas of constraint, procedural generation can massively reduce file sizes, or eliminate the need for in-game assets completely.
When Naked Sky Entertainment’s game RoboBlitz launched on PC and Xbox 360 in 2006, Microsoft had decided that the largest downloadable game it would allow on the system was a tiny 50MB, to fit on one of their (tiny for the time) memory sticks. Yet RoboBlitz was the first game to run the then-revolutionary Unreal 3 engine. As Naked Sky’s chief technical officer Joshua Glazer told us, they used lots of tricks to get the game down to under 50MB – among them procedural generation.
“All those levels would have loaded a lot faster if it weren’t for Microsoft’s 50MB requirement, coupled with the requirement that we couldn’t have any kind of ‘install’ process that unpacked the game after download,“ says Glazer. “We procedurally generated the textures for everything except Blitz (the hero) and the baddies. Pretty much every single square inch of the levels and most of the props were procedurally textured using Allegorithmic’s cool procedural texture-gen tools. Also, all the animation was procedurally driven by physics. We also procedurally modified our sound effects on the fly so that we could get more use out of the samples we were able to squish in the package. The levels were not procedurally generated – we didn’t experiment with that until our 2010 game, Microbot.”
Naked Sky’s latest game, Scrapforce, has mostly avoided procedural generation, save for an interesting-sounding stochastic AI system. And RoboBlitz isn’t the smallest game made using procedural techniques – that prize almost certainly goes to .kkrieger, a 2004 first-person shooter that’s only 96KB in size – but it is the most impressive for its size.
HARD LIMITATIONS
Despite all these positives, there are limits to procedural generation. Firstly, it can be processor intensive. Dwarf Fortress can take several minutes to get a game going, despite only displaying its graphics in ASCII. Gamers who are used to the rapid loading of mobile games might find this frustrating. And it’s power-hungry too, as Cook explains. “If you’re generating things like stories, or special abilities for an RPG, you want to know that they make sense, that they’re balanced, that they’re fun. That’s an extremely difficult thing to test, and it often needs a lot of playtesting – and that means more computing power.”
Another problem is that procedural generation works best in clearly delineated areas. Because it’s so hard to define things like ‘beauty’ or ‘fun’, procedural generation often stays in areas where it knows it can perform. In those areas, it’s commonplace. “Procedural art systems for trees and rocks are really popular – you probably don’t even notice the games they’re used in half the time,” says Cook. “People don’t really notice if a branch is slightly out of place, and trees follow quite clear natural rules that we can give to a computer, so it’s a perfect use-case.” For example, SpeedTree, the games industry standard tree generator, has appeared in nearly every AAA game for the last few years – The Witcher 3, Shadow of Mordor, Destiny, Far Cry 4…
Cook explains that when using “generative software in music, art or films, we mostly use it to target things where imperfections are hard to see (abstract art instead of portraiture), things that aren’t the focus of attention (crowd scenes instead of big centrepieces), and preferably things where there are clear patterns and rules (I think electronic music benefits here).” Games, however, often allow procedural content to be closely scrutinised, increasing the quality threshold hugely.
The game combining all these elements is Dwarf Fortress, currently the high point of procedurally generated games. “One of the strengths of Dwarf Fortress – a huge inspiration for Clockwork Empires – was that everything is procedurally generated,” says Vining. “History, terrain, monsters, gods. The newer stuff in the upcoming Dwarf Fortress patch, with procedurally generated libraries full of procedurally generated books and poetic forms, is completely insane in the best way possible.”
THE APEX
Developed by two brothers, Tarn and Zach Adams, Dwarf Fortress procedurally generates almost everything you could imagine. According to Tarn, in their previous game, Slaves to Armok, “you could zoom in on your character, and it’d tell you how curly his leg hairs were, and the melting and flash points of various materials. It was insane.”Dwarf Fortress goes further.
The entire world and its history is generated, epoch by epoch, before you start playing, with kingdoms rising, dark gods falling, and magical items being lost for centuries. Then it generates the landscape and terrain you’re in, the dwarves you’re nominally in command of, the local flora and fauna, and a hundred other things, including art and poetry that references in-game events. Dwarf Fortress is so impressive and rigorous, but also insane, that people have built working computers inside it. That’s right, a virtual machine running inside a game.
MichaelCook’sAngelinasimulation can’t claim to be as complex as Dwarf Fortress, but it’s increasing comprehensiveness in a different direction. That’s because Angelina (www.gamesbyangelina.org) is an experiment in computers designing and evaluating video games autonomously. “The games Angelina produces aren’t procedural,” he explains. “All the generation happens during the design of the game, but Angelina releases static games like you or I might if we made a game. It’s all about how we can procedurally generate a game design, ideas, art, mechanics, music and put it all together, rather than generating what happens in the player’s computer when they’re playing.”
ANGEL WITH A BROKEN WING
“Angelina is interested in generating everything in a game, which means we have to tackle the hardest problems for procedural generation, like aesthetics and emotions, as well as the more rule-based stuff like generating mazes and rocks. I want Angelina to generate program code, and game mechanics, and systems of meaning so it can tell you a story with its games or make you think.”
These two types of comprehensiveness – Dwarf Fortress’s obsessive inclusiveness and Angelina’s procedural game design – are core to the story here. Games, and their successor, VR (see “Generating Escapism”), are a universal experience. They can contain everything else in the world, so they take in every other part of generative art, if everything’s not to be handcrafted at great expense. The only way to do that, without us all becoming developers, is procedural generation.
GENERATING ESCAPISM
Procedural game design academic Michael Cook said something very interesting. “Depending on how VR designers want to take the tech, procedural generation could be the defining technology of the VR age.” As we’ve explored in previous months, AR and VR are taking the lessons from games and growing them into whole new fields – Oculus is even employing ex-Pixar staff to make VR kids movies.
Of course, like AAA games, VR has to be convincing to work, and the hardware can’t do it alone. It needs tons of convincing content – a lot more than any game, because it has to be all around the player and in great detail. Procedural generation might not just be helpful in creating these worlds and objects, it seems essential.
Valve and HTC’s Vive system adds even more complexity, given that it uses extra cameras to map the dimensions of the room. That means shifting the size and shape of the rooms you’re experiencing through the device, which means the designers will have to create the game with variable locations for cues and objects. Your fantasy tavern might just be a few feet across and need a keg in the middle to hide your coffee table, or it might be very big and absolutely clear.
Cook has ideas on how this will be dealt with. “The way we’ll solve these problems, I believe, is by having procedural generation systems that are intelligent enough to redesign games to fit inside your room. So the game designer specifies what needs to be in this tavern – a bar, a table, a door – and then your HTC Vive automatically detects where these things should go.
“It generates the layout of the room to make sure you don’t walk into that expensive vase you have in the corner, or accidentally put your knee through the TV. To me, VR is an amazing new environment to do procedural generation in.”
DREAMS OF THE DEEP
There are ways other than procedural generation and handcrafting to generate content. The most spectacular in recent weeks has been the frankly terrifying and hallucinatory images coming out of the Google Deep Dream project.
This project works by starting with a neural network. These are typically piles of 10–30 stacked layers of artificial neurons. The team starts training the neural network, using millions of examples, and then adjusts the network parameters gradually over time so that fidelity isn’t lost, but random noise is.
Each layer of the network extracts higher level features than the previous one, until the final layer decides what the image looks like. So, the first level might detect edges, then intermediate layers might detect recognisable shapes, while the final layers assemble them into complete interpretations. This is exactly how Google Photo works, too.
The reason these horror pictures have been going around, of people with hundreds of dog’s eyes making up their face, is that this particular neural network, called Deep Dream, has been fed data from ImageNet, a database of 14 million human-labelled images. But Google only uses a small sample of the database, which contained “fine-grained classification on 120 dog sub-classes”.
It’s the opposite ofMandelbrot’simages. Instead of seeing the same pattern as we go deeper into the image, the computer sees the same pattern everywhere it looks. Sadly, the pattern happens to be that of a dog.