source/a16z
Compile/Nick
Much of the early discussion of the generative AI revolution in games focused on how AI tools could increase the efficiency of game creators, allowing games to be produced faster and on a larger scale than before. In the long run, we believe AI will not only change the way games are created, but the very nature of games themselves.
All the time, AI is helping to generate new forms of games. From procedurally generated dungeons in Rogue (1980), to finite state machines in Half-Life (1998), to AI game directors in Left 4 Dead (2008). More recently, advances in deep learning technology have further changed the game by enabling computers to generate new content based on user prompts and large data sets.
It’s still early days, but we’re already seeing some interesting areas of AI-driven gaming, including generative agents, personalization, AI storytelling, dynamic worlds, and AI co-pilots. If successful, these systems could be combined to create emerging AI games that retain loyal players.
generative agent
Pioneered by Maxis' SimCity in 1989, the simulation game allows players to build and manage a virtual city. Today, the most popular simulation game is The Sims, where more than 70 million players worldwide manage virtual people known as "sims" and let them go about their daily lives. Designer Will Wright once described The Sims as an "interactive dollhouse."
Generative AI can greatly advance the development of simulation games by making agents more realistic through emerging social behaviors driven by large language models (LLMs).
Earlier this year, a research team from Stanford University and Google published a paper showing how to apply LLM to agents in games. Led by doctoral student Joon Sung Park, the research team incorporated 25 Sims-like agents into a pixel art sandbox world whose behavior was determined by ChatGPT and an architecture that extends LLM to use natural language to store a complete record of an agent's experience, synthesize these memories into higher-level reflection, and dynamically retrieve them to plan behavior.
These results are an excellent preview of the potential future of simulation gaming. Starting with a user-specified suggestion that an agent wants to host a Valentine's Day party, the agents independently distribute party invitations, form new friendships, invite each other on dates, and coordinate to arrive at the party on time two days later.
This behavior is possible because LLMs are trained on social network data, so their models include the fundamentals of how humans talk to each other and behave in various social contexts. And in interactive digital environments like analog games, these responses can be triggered to create lifelike behavior.
From the player's perspective, the end result is a more immersive gaming experience. Much of the fun of playing The Sims or colony sim RimWorld comes from the unexpected happening. With agent behavior in social networks, we may see simulation games that not only showcase the imagination of game designers, but also reflect the unpredictability of human society. Watching these sims can provide as much entertainment as watching the next generation of The Truman Show in a way that's not possible with today's pre-produced TV or movies.
The agents themselves can also be personalized, drawing on our imaginative aspirations for a "Dollhouse" game. Players can design an ideal agent based on themselves or fictional characters. "Ready Player Me" allows users to generate their own 3D avatar by taking a selfie and import the avatar into more than 9000 games/applications. AI character platforms Character.ai, InWorld, and Convai can create custom NPCs with their own backstories, personalities, and behavioral controls.
With natural language capabilities, the way we interact with agents has also been expanded. Today, developers can use Eleven Labs' text-to-speech models to generate realistic voices for their agents. Convai recently partnered with Nvidia for a well-publicized demo in which players could engage in a natural voice conversation with an AI ramen chef NPC, with the dialogue and matching facial expressions generated in real time. AI companion app Replika already allows users to chat with their mates via voice, video and AR/VR. In the future, one can imagine a simulation game where players can stay in touch with their agents via phone or video chat while traveling, and then click into a more immersive game when they return to their computer.
However, there are still many challenges to be solved before we can see a fully generated version of The Sims. The training data for LLMs has inherent biases that may be reflected in agent behavior. 24x7 real-time service games The cost of running large-scale simulations may not be economically viable, running 25 agents in 2 days would cost the research team thousands of dollars in computing. Efforts to offload model workloads to devices are promising but still relatively early. We may also need to develop new norms around quasi-social relationships with agents.
But one thing is clear, there is a huge demand for generative agents right now. In our recent survey, 61% of game studios plan to experiment with AI NPCs. We believe that AI companions will soon become commonplace as agents enter our everyday social spheres. Simulation games provide a digital sandbox in which we can interact with our favorite AI companions in fun and unpredictable ways. In the long run, the nature of simulation games is likely to change, with these agents not just toys, but potential friends, family members, colleagues, advisors and even lovers.
Personalization
The ultimate goal of a personalized game is to provide each player with a unique gaming experience. For example, let's start with character creation - from the original Dungeons & Dragons board game to Mihoyo's Genshin Impact, character creation has been the backbone of nearly every role-playing game (RPG). Most RPGs allow the player to choose from preset options to customize appearance, gender, class, etc. So how do you go beyond presets to generate a unique character for each player and gameplay? A personalized character builder combining LLM with a text-to-image diffusion model enables this.
Spellbrush's Arrowmancer is an RPG powered by the company's custom GAN-based animation model. In Arrowmancer, players can generate a complete set of unique anime characters, including art and combat abilities. This personalization is also part of its monetization system, with players importing AI-created characters into custom gacha banners, where they can earn duplicate characters to strengthen their ranks.
Personalization also extends to in-game items. For example, AI can help generate unique weapons and armor that are only available to players who complete certain tasks. Azra Games built an AI-powered asset pipeline to quickly ideate and generate a vast library of in-game and world items, paving the way for a more diverse gaming experience. Renowned AAA developer Activision Blizzard built the Blizzard Diffusion system, a replica of image generator Stable Diffusion, to help generate concept art for various characters and outfits.
In-game text and dialogue can also be personalized. Emblems in the world can reflect some kind of title or status the player has achieved. NPCs can be set up as LLM agents with unique personalities that adapt to the player's behavior. For example, dialogue can change based on the player's past behavior with the agent. We've seen this concept implemented successfully in a triple-A game, and Monolith's Shadow of Mordor has a revenge system that dynamically creates interesting backstories for villains based on player actions. These personalization elements make every gaming experience unique.
Game publisher Ubisoft recently revealed Ghostwriter, a conversational tool powered by LLMs. Today, publishers use the tool to automatically generate dialogue that helps simulate the living world around players.
From the player's point of view, AI adds to the immersion and playability of the game. The enduring popularity of role-playing mods in immersive open-world games like Skyrim and Grand Theft Auto V demonstrates a latent need for personalized stories. Even today, GTA V consistently has a higher player count on role-playing servers than the original game. We believe that in the future, personalization systems will become an integral real-time operational tool for attracting and retaining players across all games.
AI narrative
Of course, there's more to a good game than characters and dialogue. Another interesting scenario is leveraging generative AI to tell better, more personal stories.
Dungeons & Dragons is the granddaddy of personalized storytelling in games, in which a person known as a "dungeon master" prepares to tell a story to a group of friends who each play a role in the story. The resulting story is part improv drama, part RPG, meaning each playthrough is unique. In a sign of the need for personalized storytelling, Dungeons & Dragons is more popular than ever today, with digital and analogue sales hitting record highs.
Today, many companies are applying LLM to the Dungeons & Dragons story mode. The opportunity here is for players to spend their time in their favorite player-created or IP universes, guided by an infinitely patient AI storyteller. Launched in 2019, Latitude's AI Dungeon is an open-ended, text-based adventure game in which the AI plays the dungeon master. Users have also fine-tuned OpenAI's GPT-4 version to play Dungeons & Dragons with promising results. Character.AI's text adventure game is one of the app's most popular modes.
Hidden Door goes a step further by training a machine learning model on a specific set of source material (such as The Wizard of Oz) and letting players adventure within a given IP universe. In this way, Hidden Door worked with the intellectual property owner to enable a new, interactive form of brand extension. As soon as fans finish watching a movie or book, they can continue their adventures in their favorite worlds through custom events similar to Dungeons & Dragons. Demand for the fan experience is booming, with Archiveofourown.org and Wattpad, the two largest online fan fiction repositories, receiving more than 354 million and 146 million website visits, respectively, in May alone.
NovelAI developed its own LLM Clio, using it to tell stories in a sandbox mode to help human writers overcome writing block problems. For the most discerning writers, NovelAI lets users fine-tune Clio based on their own work, or even that of famous authors like HP Lovecraft or Jules Verne.
It's worth noting that there are many hurdles before AI story production is fully ready. Today, making a good AI storyteller requires a lot of human rule-setting to create the narrative lines that define a good story. Memory and coherence are very important, the storyteller needs to remember what happened earlier in the story and be consistent in fact and style. Interpretability remains a challenge for much closed-source code that operates as a black box, and game designers need to understand how systems behave in order to improve the gaming experience.
In overcoming these hurdles, however, AI has become the co-pilot of human storytellers. Today, millions of writers use ChatGPT to inspire their stories. Entertainment studio Scriptic has brought together DALL-E, ChatGPT, Midjourney, Eleven Labs, and Runway with a human editorial team to create an interactive, choose-your-own-adventure show, now streaming on Netflix.
dynamic world building
While text-based stories are popular, many players are also eager to see their stories brought to life visually. Perhaps one of the biggest opportunities for generative AI in gaming is to help create living worlds that players spend countless hours immersing themselves in.
The ultimate vision is to be able to generate levels and content in real time as the player progresses through the game. The "Mind Game" in the science fiction novel "Ender's Game" (Ender's Game) is a typical example of this kind of game. The Mind Game is an AI-guided game that adapts in real time to each student's interests, with the game world constantly changing based on the student's behavior and any other mental information the AI infers.
Probably the closest thing to a "mind game" today is Valve's Left 4 Dead series of games, which utilize AI guidance to dynamically adjust gameplay pacing and difficulty. Instead of setting the spawn point of enemies (zombies), the AI director places zombies in different positions according to each player's status, skills and position, creating a unique experience in each game. The director also uses dynamic visual effects and music to create the atmosphere of the game. Valve founder Gabe Newell calls this system "programmed storytelling." EA's critically acclaimed Dead Space remake uses a variant of the AI director system to take the horror to extremes.
While this may seem like the stuff of science fiction today, one day, with improved generative models and the availability of enough computation and data, we may have an AI director who can create not just scares, but the world itself.
It's worth noting that the concept of machine-generated levels in games isn't new. From Supergiant's Hades to Blizzard's Diablo to Mojang's Minecraft, many of today's most popular games use procedural generation, which uses equations and sets of rules run by human designers to randomly create levels that are different every time. A complete set of software libraries has been established to assist program generation. Unity's SpeedTree helps developers generate the virtual foliage you might have seen in the forests of Pandora in Avatar or the landscapes of Elden Ring.
A game could combine a procedural asset generator with LLM in the user interface. The game "Townscaper" uses a procedural system that only needs the player to input two pieces of information (the position and color of the blocks), and it can be quickly transformed into a gorgeous townscape. Imagine adding LLM's Townscaper to the user interface to help players iterate more refined and exquisite works through natural language prompts.
Many developers are also excited about the potential of using machine learning to enhance program generation. One day, designers could iteratively generate viable first drafts of levels using models trained on existing levels with similar styles. Earlier this year, Shyam Sudhakaran led a team at the University of Copenhagen that created MarioGPT — a GPT2 tool that generates Super Mario levels using a model trained on the original levels from Super Mario 1 and 2. Academic research in this area has been going on for some time, including a 2018 project using generative adversarial networks (GANs) to design levels for the first-person shooter DOOM.
Generative models, used in conjunction with procedural systems, can greatly speed up asset creation. Artists are already using text-to-image diffusion models for AI-assisted concept art and storyboarding. In this blog post, mainframe visual effects supervisor Jussi Kemppainen describes how he built the world and characters for a 2.5D adventure game with the help of Midjourney and Adobe Firefly.
3D generative techniques have also been heavily researched. Luma leverages Neural Radiation Fields (NeRFs) to allow consumers to build photorealistic 3D assets from 2D images captured on an iPhone. Kaedim uses a combination of AI and human quality control to create production-ready 3D meshes and is currently used by more than 225 game developers. CSM recently released a proprietary model that can generate 3D models from video and images.
Real-time world building with AI models is what matters in the long run. In our opinion, in the future, the entire game will no longer need to be rendered, but will be generated at runtime using neural networks. Nvidia's DLSS technology can already generate new high-resolution game frames on the fly using consumer-grade GPUs. Maybe someday, you'll be able to hit the "interact" button in a Netflix movie and step into a world where every scene is generated on the fly and tailored to the player. In the future, games will be no different from movies.
It's worth noting that a dynamically generated world alone isn't enough to make a good game, as evidenced by the review for No Man's Sky. The promise of dynamic worlds lies in its combination with other game systems (personalization, generative agents, etc.) to open novel forms of storytelling. After all, the most compelling part of "Mind Games" is how it molds itself for Ed, not the world itself.
AI "co-pilot"
While we’ve previously covered the use of generative agents in simulated games, there is another emerging use case where AI acts as a gaming co-pilot, guiding us through the game, and in some cases even fighting alongside us.
For players getting started in complex games, the role of the AI co-pilot is immeasurable. For example, a UGC sandbox game like Minecraft, Roblox, or Rec Room is a rich environment where players can build almost anything they can imagine given the right materials and skills. But the learning threshold is very high, and it is not easy for most players to find a way to get started.
The AI co-pilot can make any player a master builder in UGC games, providing step-by-step instructions based on text prompts or pictures, and guiding players through mistakes. A good point of reference is the concept of "master builders" in the Lego world, these rare beings who have the gift of being able to see the blueprints of any creation they can imagine when needed.
Microsoft has begun developing an AI-assisted system for Minecraft that uses DALL-E and Github Copilot to allow players to inject assets and logic into Minecraft sessions through natural language prompts. Roblox is actively integrating artificial intelligence generation tools into the Roblox platform with the mission of enabling "every user to be a creator." From coding with Github Copilot to writing with ChatGPT, the effectiveness of AI copilots for co-creation has been proven in many fields.
In addition to co-authoring, an LLM trained on human game data should be able to understand how to behave in various games. With proper integration, the agent can act as a partner when the player's friends cannot participate, or as the other side in head-to-head games such as FIFA and NBA 2k. Such an agent can participate in the game at any time, whether it wins or loses, it is amiable and will not blame the player. Fine-tuned based on our individual play histories, such agents could vastly outperform existing bots, playing exactly the way we do, or playing in complementary ways.
Similar projects have been successfully run in constrained environments. The popular racing game Forza has developed a "Drivatar" system that uses machine learning to create an AI driver for each human player that mimics their driving behavior. Drivatars are uploaded to the cloud, and when the human partner is offline, Drivatars can be invoked to race against other players and even earn victory points. Google's DeepMind's AlphaStar was trained on a "200-year-old" StarCraft II game dataset to create an agent that can play against and beat human e-sports pros.
As a game mechanic, the AI co-pilot can even create entirely new game modes. Imagine Fortnite, but each player has a "master builder" wand that can instantly build sniper towers or flaming boulders with prompts. In this game mode, the outcome may depend more on what the wand does (hint) than on the ability to aim the gun.
The perfect AI "partner" in games has been a memorable part of many popular game franchises. Examples include Cortana in the Halo universe, Elle in The Last of Us, or Elizabeth in BioShock Infinite. Beating up computer bots never goes out of style for competitive gaming -- from frying aliens in Space Invaders to combat stomping in StarCraft, which eventually morphed into its own game mode, Co-op Commander.
As gaming evolves into the next generation of social networking, we expect the AI co-pilot to play an increasingly important social role. It’s been well documented that adding social features can increase game stickiness, with retention rates up to 5x higher for players with friends. In our opinion, every game in the future will have an AI co-pilot.
in conclusion
We're still in the early days when it comes to applying AI to games, and there are many legal, ethical, and technical hurdles that need to be resolved before these ideas can be brought to life. Currently, legal ownership and copyright protection for games using AI-generated assets is largely unclear unless developers can prove ownership of all data used to train models. This makes it difficult for owners of existing licensed intellectual property to use third-party AI models in their production pipelines.
How to compensate the original authors, artists, and creators behind the training data is also a major issue. The challenge is that most AI models are trained on public data on the internet, most of which are copyrighted works. In some cases, users have even been able to reproduce an artist's style using generative models. It is still early days, and the issue of compensation for content creators needs to be properly resolved.
Most generative models are currently too expensive to run in the cloud at global scale 24/7, which is required for modern game operations. To keep costs down, application developers may need to find ways to offload model workloads to end-user devices, but this will take time.
However, it is now clear that game developers and players have a lot of interest in generative AI for games. While there’s also a lot of hype, we’re excited to see how many talented teams in this space are working overtime to build innovative products.
The opportunity isn't just to make existing games faster and cheaper, but to create a whole new kind of AI game that wasn't possible before. We don't yet know what form these games will take, but we do know that the history of the games industry has been one of technology enabling new forms of play. With systems like generative agents, personalization, AI storytelling, dynamic world building, and AI co-pilots, we may be on the verge of seeing the first "never-ending" games created by AI developers.
Gyro Finance Contact Information
Business Cooperation|Contribution:
Xiao Huang (WeChat ID 18925291949)
Ning (WeChat ID 13631579042)
recommended reading
