Superhuman League of Legends AI by 2027 (Part 4 - Brokenwings: 5v5)

Table of Contents

• Table of Contents
• Overview
• Why are we going back to Brokenwings?
• Phase 1: Making 5v5 fast
  • Profiling Numba and long compilation times
  • Rewrite it in Rust
  • Why the simulator is fast
• Phase 2: Training 5v5 agents
  • It won’t fight
  • It won’t end
  • It farms, just as it was asked to…
  • Recalling
  • So how good are they?
• Phase 3: Why won’t they end the game?
  • The suspects
  • Peering inside the agents
  • The turn
  • The culprits
  • Before and after
  • The payoff
• So where do we go from here?
• Appendix
  • Observation and action space
    • What a hero sees
    • What a hero can do
    • What the agent does not control
    • Two rules that exist only in training
    • How it changed
  • Reward weights
    • Individual rewards
    • Team rewards
    • What happens to the numbers afterwards
    • The two signs
  • Network architecture
    • How it is trained
    • Next to OpenAI Five
  • How the ladder was built
    • The games
    • The rating
    • How to read the numbers
    • Two engines, one ladder
    • What it cannot tell you
  • Roster Selection
  • Training Log

Overview

In the last post, we focused on porting our findings from Brokenwings to modern League of Legends (patch 26.19). We managed to create a faster than real-time simulator for League of Legends which had similar throughput to OpenAI Five.

Now we want to graduate up to full 5v5 games. However before porting over more of modern League of Legends to our gameserver, we instead want to validate two things:

1. Scaling up fastsim to the full 5v5 game mode
2. Determining if our current RL methods can handle the full 5v5 game mode given that its game-tree complexity is far higher than a simple mid lane Ezreal 1v1

Why are we going back to Brokenwings?

At this stage we’re confident we can train 1v1 agents, it’s just a matter of compute and time. However, have we actually tested our core hypothesis yet? Do we have any idea how much compute is required to train 5v5 agents for League of Legends and do we know if our fastsim throughput gains hold for the full game, not just a 1v1 minigame involving a few minion waves, two champions and two towers?

To answer this, we’re going back to brokenwings as it is a faithful enough implementation of the 2011 game server (patch 1.0.0.126) and helps us answer our fundamental questions. Unlike the modern game server where we don’t have the source code, brokenwings is open source which means we don’t have to worry about the additional overhead of validating the fidelity of our game servers implementation. We only have to ensure that we’ve transpiled the existing game server faithfully into a more efficient implementation.

This journey led to three distinct phases, over one week:

Phase 1: Making 5v5 fast

This stage involved making the 5v5 fastsim version of Brokenwings as fast as our GH200 hardware could possibly run. In the end, here is how much each method improved environment throughput, measured in years of game experience simulated per day:

To start with, our fully working Brokenwings game server with our custom RL bridge could already run full games. Using the same approach as OpenAI Five we were able to simulate games and hook them. However this means our only speedup was running the game on more CPU cores, which to reach the same throughput as OpenAI Five or AlphaStar would be incredibly expensive.

Earlier in September, we focused on speeding up the game server by stepping each game tick instead of running at 30hz. This involved finding every line of code in the C# engine which relied on wall-clock time and instead setting it to use the game server’s per step provided time.

In the last post we ported enough of the game engine for an Ezreal 1v1 and when combined with our training infrastructure managed to achieve up to 1.95M agent decisions per second.

Now we wanted to do this for the full 5v5 game and set a target of 24 hours for the initial port. With Brokenwings original source code, we had a perfect oracle to validate behaviour exactly.

Our first approach involved porting the game server to a python reference and creating the Numba implementation in parallel - by directly transpiling functions from the C# game server and validating behaviour.

For context, Numba is a Python package which wraps Python source code and compiles it just-in-time (JIT) into machine code using LLVM. This allows you to greatly improve the performance of your Python code without fully translating it into another language. Now the box simulates 10,400 seconds (~2:53 hrs) for one wall-clock second.

In total we simulate 12 years worth of game play per day (during training) in contrast to OpenAI Five’s 180 years a day which means that we’re ~15x off. However, they use 128,000 CPU cores (likely vCPUs) whereas we use 64 GH200 neoverse v2 CPU cores.

Profiling Numba and long compilation times

At this stage our simulator already achieved high throughput but we ran into new problems. Every time we wanted to add new functionality to our Numba simulator, we were forced to re-compile the entire simulator because recursion broke our cache. We fixed the cache by breaking recursion, which took a restart from 30 minutes to 11 seconds but any big engine change still cost a full recompile. Cold compilation took between 20-40 minutes per each change. During this we also profiled the bottlenecks within the Numba fastsim itself:

One major step stands out, reference counting. This is used to track if arrays are still in use for garbage collection. The issue with our fastsim is that the whole game state lives in one state object holding many arrays. Every time a function reads an array out of that object, Numba increments a counter, and decrements it when the function returns. Our hot helpers are tiny and called thousands of times a tick, so the counting cost more than the work. And nothing was ever freed: the arrays live for the whole run.

Rewrite it in Rust

Bottleneck

the compiler's bookkeeping

Lesson

profile first; the rewrite was the conclusion, not the plan.

“Rewrite it in Rust” is usually a punchline. Here the profile made the case for us as 59% of the engine was spent counting references to arrays that were never freed, and every attempt to fix that inside Numba cost a half-hour compile.

The Rust port re-used the Numba arrays for development speed and also reads and writes the same state arrays as the Numba engine and exposes one step function to Python, so everything around it stayed as it was, and Numba became the oracle. A lead agent spent 25 minutes on the groundwork: a generated description of the state layout, a rulebook for numerics (the same float32 operation order, no fused multiply-add, no fast-math), and a harness that steps both engines from the same recorded state and reports the first array, index and tick where they differ.

Then the engine was cut into nine families (navigation and vision, movement, casting, AI, rules, damage, the kit core, the scripting API, jungle and kits) with one translator agent each.

The speedup is larger than the 2.3× that removing reference counting alone would have given. With the counter updates gone, the compiler could finally optimise the code around them. A rebuild after a change takes minutes rather than half an hour, and an overnight tuning pass on the trainer took end-to-end training from 0.33M to about 1.2M agent-steps a second. Now we could run 40–44 game-years a day on one box, against roughly 80 for OpenAI Five’s Rerun on 51,200 cores.

Why the simulator is fast

On the same 64-core GH200, the real C# game server manages about 2,500 decisions a second across the whole box, while fastsim, the Rust engine, sustains about 125,000 in training. Most of that gap is not faster arithmetic: the server gives every game its own process and talks to the policy in text, while fastsim keeps every game as rows in shared arrays and makes one call and one forward pass per decision for all of them.

Phase 2: Training 5v5 agents

The setup is deliberately close to OpenAI Five’s in June 2018, restrictions included. Two fixed teams of five (Ezreal, Garen, Ashe, Veigar, Malphite against Caitlyn, Warwick, Nunu, Master Yi, Ryze - full reasoning here), no summoner spells, no wards. One network drives all ten heroes, told apart by a champion one-hot. Each hero sees about 2,900 numbers per decision under fog of war and acts every 133 ms - 7.5 Hz. Training is PPO with an LSTM: 4,096 games at once, 64 decisions per rollout, 2.6 million agent-steps per update, the two teams swapping sides every 1,200 rollouts.

It won’t fight

Bottleneck

what ends an episode

Lesson

if doing nothing pays zero and fighting can pay less, self-play will do nothing.

The first thing we ran on the new simulator was a smoke test. Both teams dropped next to each other in mid lane, levels 6 to 11, with a 40-second cap. If ten heroes sharing one network could learn anything together, a team fight seemed the fastest way to find out.

They learned something within the hour. At rollout 50 the games look like a team fight should as one side is wiped in under 20 seconds, every round. By rollout 100 nearly half the games were running out the clock, and by rollout 150 all of them were. Kills per game went from six to zero, and the value loss fell to 0.001, because there was nothing left to predict. A recording from rollout 375 shows one straggler being picked off and then ten heroes standing apart in mid lane until time runs out.

Nothing was broken. The reward was zero-sum with shaping for health, so running out the clock paid exactly zero to both teams. A fight paid about −1.3 per hero to whichever side had the weaker composition, and the two rosters are far from equal: playing random actions, one of them wins 83% of the time. It takes two to fight, and in an open lane the weaker side can always walk away. Not fighting is a Nash equilibrium of the reward we wrote, and PPO found it in about 100 rollouts.

We tried to price the draw. A penalty of 1.0 for hitting the cap moved the draw rate from 100% to 88%. Doubling it moved it back to 93%. A penalty only works on the weaker team once it exceeds what that team expects to lose by fighting, and at that point we would be paying it to die.

This is the third time the series has met this equilibrium. In Part 2 the mirror match settled into a standoff with both sides ending every episode on 188 HP. In Part 3, two equal Ezreals could always walk away, and the fix was a wall around the arena. With ten heroes the fix is the game itself. We put minions in the lanes and towers behind them, and standing still starts to lose. At 18:02 the same evening we stopped tuning the fight and switched to full games.

It won’t end

Bottleneck

the clock: a 12-minute cap and a 27-second horizon

Lesson

an agent can only learn to finish a game if the finish fits inside its episodes and its discount.

Full games were supposed to make avoidance expensive. First they had to run. The game task trained at 0.04M agent-steps a second, twenty times slower than the benchmark, and the cause was not the engine: a path across the map costs about 100 times a local one, and an untrained policy orders them constantly. Limiting move targets to 1,500 units on the agent’s side, with no change to any rule, took it to 0.2M, and three bit-identical engine fixes took it to 0.33M.

Then they had to matter. Six-minute games from level 1 never saw a turret fall: at the first cap, all 4,096 games ended with zero turrets down. So most games now start in the middle: 5 to 20 minutes in, heroes pre-levelled and in lane, outer turrets already damaged. With the cap raised to 12 minutes, the first games in the project ended with a nexus kill at 22:32 that night.

Almost none of the others did. 98% of games still hit the cap, and a reward audit showed what that meant:

The policy was being paid to farm and push, and almost never to win. We tried paying capped games by structure lead. At half the win reward the policy stopped fighting and sat on its leads: kills per game fell from 24 to 18 and the nexus rate halved. At 0.3 it fought again and kept most of the gain.

Two measurements then put the result in proportion. With idle heroes, the minion waves alone take 6.8 turrets a game, so about 92% of the turrets we were crediting to the policy were falling anyway. And against heroes that do nothing, the policy took the nexus in 98% of games given 30 minutes, and in 0% given the 12 it trained on. It could finish. Its episodes never lasted long enough for it to be paid for finishing, and with γ = 0.995 at 7.5 decisions a second, anything more than about 27 seconds away was worth nearly nothing to it anyway.

By rollout 6,750 the ladder had been flat for four hours. So we changed the clock: an 18-minute cap, a 67-second horizon (γ = 0.998), a quarter of games starting as a siege with the base already open, damage to the nexus paid directly, and the win reward doubled. The plateau broke within the hour. Nexus kills from a normal start went from 1–3% of games to 8%, and in a 24-minute matchup 30% of games were decided, against about 5% at 12 minutes.

More time, more finished games. It looked like a problem we could keep solving the same way.

It farms, just as it was asked to…

Bottleneck

the reward pays for farming three times over

Lesson

you get what you pay for, including the parts you forgot you were paying for.

When we paused at rollout 7,350 the agents barely farmed. They took 38% of the available last hits, touched the blue buff in 7 games out of 100, and spent 1.9% of their time grouped as five. Forty thousand rollouts later they were different animals:

The jungle is the striking one. By the end a game has about 67 monster kills in it, with the first camp taken around two and a half minutes in. Nobody told the agents who should jungle or when, and their answer is not the one a player would give. The two champions we picked as junglers mostly stay in lane. The camps go to the ranged and caster heroes, who step out of an empty lane to the nearest one and step back:

Lanes are just as unorthodox. There is no one-one-two split: in the first four minutes Ezreal, Ashe, Veigar, Caitlyn and Ryze each spend about half their time mid, while Malphite goes top and Warwick goes bottom.

It would be nice to call this emergent, but it is closer to obedience. The reward pays 0.16 for a last hit on a lane minion and the same for a jungle monster, and has done since the first run. It also pays for gold and for XP, and a last hit gives both. So one minion is paid three times. A kill is paid the same way: +1, plus its gold, plus its XP. OpenAI Five’s reward has the same terms with the opposite sign, −0.16 for a last hit and −0.6 for a kill, precisely to cancel the gold and XP that come with them. Ours stacks where theirs subtracts.

Watching the long games, the result looks like low-ranked human play: ten players who have learned that farming is safe and fighting is risky, who do attempt to group, and who rarely turn a lead into a win.

Recalling

Bottleneck

an action that only pays if you keep choosing it

Lesson

when a behaviour cannot be explored, change the environment before you change the reward.

On Wednesday we added items and the Recall spell. Items were easy: a scripted buyer spends each hero’s gold whenever it stands in the fountain, so there is no new action and nothing to learn. Recall was a new action, added to the trained network by surgery (one new output with zero weights, so the policy behaved exactly as before until training moved it).

Recall is hard for a sampling policy for a simple reason. It is a channel, and any move order cancels it. An agent that picks an action every 133 ms has to pick “do nothing” dozens of times in a row to get home, and the pay-off (a full health bar, new items, and whatever those are worth over the next ten minutes) arrives long after the decision. It took four attempts.

The fourth attempt made Recall possible. It did not make it worth anything the agent could see, because with a 67-second horizon the benefit of an item is mostly out of view. The OpenAI Five paper warns about exactly this when handing a new action to the model: “the agent might quickly learn never to take a new action”. Its reward table also has a term we had skipped: 0.0006 per unit of gold “spent on items without using courier”, which pays a hero for shopping in person. We borrowed it at 0.0015, and only for gold spent while alive, so that dying and shopping at the respawn earns nothing. We also reset the Recall output so the policy would explore it afresh.

That worked. Agents now go home about every two minutes, complete 81–88% of the recalls they start, and spend 83% of their gold on trips they chose to make. On the real server, in a 45-minute game, 90 of 117 recalls completed. The checkpoint a thousand rollouts after the change beat the one before it 62% of the time.

Three of the four attempts changed what the agent could see or was allowed to do, and only the last touched the reward. The lockout and the observation were both cheaper than the fix that worked, which is why we tried them first, and both taught nothing. What the agent needed was for the commitment to be made on its behalf.

So how good are they?

Bottleneck

a ladder can only rank what its games decide

Lesson

when 95% of your games are draws, the number you are tracking is not the one you care about.

There are three ways to ask, and by Thursday morning they gave three different answers.

Against the game’s own bots: very. On Monday, at rollout 5,875 and before it had ever bought an item, the policy played 64 games against the server’s scripted bots and won 55, drew 8 and lost 1, with 45 kills to 3. Two days later, with items, a checkpoint played a full game on the real client and took the nexus in under half an hour. This was the first time the agents met anything from outside their own lineage, and it was not close.

Against their earlier selves: it depends how you count. On Thursday we paused training and played 22 checkpoints from across the run against each other: 43,264 games, each from the start, with the 18-minute cap the agents trained under. 94.5% of those games hit the cap. Count a capped game as a draw, and the whole run from rollout 2,400 to rollout 46,575, about 115 billion agent-steps, is worth 66 Elo. Break the ties by structures destroyed and then net worth, and the same games show a steady climb to 1,465. Both are true. One says the agents kept getting better at being ahead after 18 minutes. The other says they almost never beat each other.

Watching them: they do not finish. We took the best checkpoint and let ten games run to an hour. Six were still undecided at 60:00. The earliest ended at 29:08. In one, a team killed inhibitors eleven times and never took the nexus. On the real server, a 45-minute game ended with both nexuses at full health. It looks like low-ranked human play: farming is safe, fighting is risky, and nobody knows how to group and close.

So we removed the cap altogether and ran 1,024 games to see how long they would go.

For comparison, we ran the same thing with ten heroes that never move. The minion waves alone end that game in 36 minutes. Our agents, after 128 billion agent-steps of practice, take nearly twice as long as nobody at all, and one game in ten they prevent it from ending.

That is not a weak agent. A weak agent loses. This is something else, and Phase 3 is the story of finding out what.

Phase 3: Why won’t they end the game?

The suspects

Bottleneck

too many good explanations

Lesson

when every hypothesis is plausible, none of them has been tested.

We had agents that farm, fight, push and go home to shop, and games that do not end. There was no shortage of reasons, and each came with a precedent in the OpenAI Five paper.

The horizon. With γ = 0.998 at 7.5 decisions a second, the agent discounts the future with a horizon of 67 seconds. A win five minutes away is worth 1% of its value today: 0.11 out of 10. An inhibitor that has just respawned is worth 2, right now, every time. Five trained with a horizon of 180 seconds and later 360.

The cap. Every training episode ends by minute 18, and starts somewhere between minute 5 and 32 of a game. Nearly all the real endings the critic ever saw came from the quarter of games that start as a siege. An even game played through to a finish was not in the data. Five had no cap at all.

The reward. Farming is paid three times over, as we saw. Fighting is too. A capped game is settled by structure lead, so “ahead at 18 minutes” is itself a paid objective. Against all that, the win is worth 10. Five paid −0.16 for a last hit and −0.6 for a kill.

Team spirit. Each hero keeps 70% of its own reward and shares 30% with the team. Five started there and annealed towards sharing everything, because closing a game is a team action and farming is not.

The model. The LSTM is trained on windows of 64 decisions, eight and a half seconds. Perhaps it cannot hold a plan for longer than that, and a siege is a long plan.

All of these were plausible reasons why our agents were constantly running the clock, but we decided to take a step back.

Before launching, we stopped to look inside the agents we already had.

Peering inside the agents

Bottleneck

we knew what they did, not why

Lesson

ask the network before you change it.

Then four studies ran in parallel on the latest checkpoint, each with one question and each ending, as in Part 3, in a decision.

How much memory does it use? We wiped one team’s actor memory every N seconds and let it play the other team, about 590 games per setting. Wiped every decision, it loses 94% of games. Wiped every second it scores 0.21; every five seconds, 0.41; every ten seconds, 0.46, which is the same as never. The actor uses the last five to ten seconds and nothing more, and 97 of its 512 memory units never fire. The critic is different: wiping it every two minutes nearly halves how well it predicts returns.

Decision: a bigger LSTM is not worth doing. Memory length, not size, is what the actor lacks.

What does it know that it cannot see? Very little. We trained probes to read a hidden enemy’s position from the LSTM state. Adding the state to a plain record of “last seen here, this long ago” makes the guess worse. It does not track enemy cooldowns in the fog, and it does not know its own previous action. Worse, it loses what it can see: an enemy hero’s position can be read from that unit’s encoding with R² 0.92, from the pooled heroes with 0.70, and from the LSTM state with 0.37. The max-pool that makes the network indifferent to unit order also makes it forget which hero was which.

Decision: pass the nine other heroes to the LSTM unpooled, feed back the previous action, and stop capping the "last seen" timer at ten seconds.

What does the critic believe? At its own 67-second horizon it explains 44% of the variance in returns. It is a poor judge of who is winning: the sign of its value predicts the winner with AUC 0.71–0.78, while the two team line-ups alone give 0.80. But a linear readout of its state predicts 180-second returns as well as it predicts 67-second ones, so the information for a longer horizon is already there. Returns at 180 seconds have nearly twice the spread.

Decision: the 180-second horizon is safe to switch on, with return normalisation added alongside it.

What triggers what? Roles come from the champion label. Give a hero its own abilities but another champion’s one-hot, and its jungle farming moves 93% of the way to the other champion’s habits. The game clock matters in one odd way: past the range seen in training, it switches Recall off. Clamping the clock changes nothing about how long games last.

Decision: rescale the clock input before lifting the cap, or the agents will stop going home in long games (by minute 60 the chance of a Recall is a hundredth of what it is early on).

The picture is the one from Parts 2 and 3 at larger scale. These are reactive agents with a few seconds of memory and a local view, and the things they lack (a plan, a map, a sense of the enemy) are the things neither the network nor the reward ever gave them. It explained a lot. It did not explain a game that lasts ten hours.

The fourth study had a second question: why do some games never end? To answer it, it built a viewer.

The turn

The fourth study had taken 256 games, found the 50 that were still running after two hours, and sorted them into two kinds. In 23, one team was camped in the enemy base, killing inhibitors as they respawned and never touching the nexus. In the other 27, nothing was happening at all: both teams had most of their turrets, nobody died, and the ten heroes spent three quarters of their time in the jungle with the nearest enemy 5,000 units away. The write-up called the second kind a standoff and described it as an equilibrium: two teams that had each decided the other was not worth attacking.

It was a good story. It matched everything we had just learned about agents that farm because farming is paid, and it was the fight task’s stalemate again at full scale. We had the launch script open.

The study came with a viewer, so we opened one of the standoff games and scrubbed to the two-hour mark, to watch what an equilibrium looks like.

There were no minions on the map.

Not few. None, in any lane, for either team. Ten heroes on an empty Summoner’s Rift, clearing jungle camps because the camps were the only things left that respawned. This meant either our visualiser was broken or fastsim itself was broken.

The culprits

Bottleneck

fidelity past the seventh minute

Lesson

a diff against the oracle cannot find a bug the oracle shares, or one that needs longer than your traces.

There were four, and between them they explain every game that would not end.

The nexus that never re-opens. The original map script has a branch to re-open the nexus when an inhibitor dies again. It can never run, because it looks up the nexus turrets in a list that dead turrets are never removed from. In 24 games that ended with one side locked, the locked side won all 24. When we emulated the intended rule, 11 of 11 locked games ended within seconds.

The pile. One corner of the minion path, next to the inhibitor, is walkable but too narrow for a minion’s body. If anything is standing there, usually a hero, minions collide, get pushed back out, walk back in, and repeat for ever, while every later wave joins them. Put a hero on that corner and 82 minions are stuck within eight minutes. The original server does the same, and the proof was in our own files: the recorded server traces we had certified against contain the same loop, up to 475 times in a row.

The pool. This one we added. The fast engine keeps minions in a fixed pool of 192. A pile fills it, and from then on no wave spawns anywhere. By minute 60 that had happened in a quarter of games, and by minute 90 in nearly nine out of ten.

The leak. We found the last one while checking the fixes, in a counter that had been ticking the whole time and that nobody read. When a minion dies its slot is reused, but the Rust port never released the dead unit’s buffs. The buff pool filled about fifteen minutes into every game, and after that new buffs were dropped: item auras, shields, slows, silences, jungle buffs, even the one that lets damage interrupt a Recall. It was not symmetric. Veigar alone held 41 dead copies of one buff. It was happening in 64% of training games.

None of this could have been caught the way we catch everything else. The simulator was bit-identical to the server on 354 traces out of 354, and the longest was under seven minutes. Two of the four bugs are in the server, so the diff was clean by construction. The other two need a long game.

Part 2 told this story once already, about a lane simulator that “matched the server on every trace we had” and hid two bugs “the short clean traces never exercised”. We wrote that down in September and walked into it again in November, with ten heroes and sixty-minute games.

Before and after

Bottleneck

none, for once

Lesson

fix the world before you retrain the agent: the same weights, on a correct game, were a different result.

The fixes were small. The nexus re-opens when an inhibitor dies again. A minion that has tried to escape the same spot twenty times stops trying and walks on. A full minion pool evicts its oldest minion, which no longer happens now that nothing piles up. A reused slot releases what the last unit left behind. The first two went into the C# server as well, in two files, so that the simulator and the game it copies agree again.

We changed nothing about the agents. Same checkpoint, same weights, played again:

Every game ends. Against ten heroes that never move, the policy used to win nine games in ten and spend three hours on the tenth; now it wins all ten in 14 to 18 minutes. The lock still happens, as it should, and now it opens: in one recorded game a nexus opens, locks again when the last inhibitor respawns at 1:10, opens when an inhibitor falls, and dies at 1:16.

The roster number is the one that stings. For four days one team had been losing its item auras, shields and crowd control a quarter of an hour into most games, and the agents had learned to play around a handicap we did not know was there. Everything we concluded about which champions are strong came from that.

And the agents are not acquitted of everything. The median game still takes 59 minutes. In two of ten recorded games the winner killed eight and eleven inhibitors before finishing. They are slow to close, for the reasons the suspects section gave. But it is a slow game now, not an endless one, and that is a problem training can fix.

The payoff

Bottleneck

finally, the agents

Lesson

rate on wins, in games that end, against something from outside the lineage.

With games that end, we could at last ask the question from Phase 2 properly. We took seventeen entrants: fourteen checkpoints from across the run, the server’s own scripted bots, a policy that acts at random and one that does nothing. A win is a dead nexus. There is no tiebreak, because there is nothing to break: of 10,752 games, not one reached the two-hour cap.

Two things in that chart were invisible a week ago.

The run never plateaued. The capped ladder had said the agents stopped improving around rollout 9,600 and that the whole run was worth 66 Elo in wins. Counted properly, strength climbs all the way: 1,251 at rollout 2,400, 1,840 at 9,600, 2,625 at 32,400, 2,966 at 46,575. The plateau was the cap.

The bots were passed almost at once. They sit between rollout 600 and rollout 1,350, about two to three billion agent-steps, or the first 3% of training. A checkpoint from the first evening beats them four games in five. Everything since is a climb with no outside reference, which is its own problem for another post.

The bot games ran on the real C# server, the rest in the simulator, and the one pairing we played on both agreed: 98.4% on the server, 98.0% in fastsim. It also made the case for the whole exercise one more time. The 10,000 simulator games took 35 minutes. The 704 server games took an hour.

Not everything on the ladder flatters us. The checkpoint we had spent the week dissecting turns out to be 292 Elo weaker than the one saved 25 rollouts later, and we do not yet know why. Every one of these agents was trained on the broken game and is being rated on the fixed one. And the answer to Phase 3’s question is both: the simulator was stopping the games from ending, and the agents are slow to end them. We had the order of blame the wrong way round.

Everything in this post ran on one GH200, but the engine itself is small. Below, the Rust engine compiled to WebAssembly and the best checkpoint on the ladder, rollout 46,575, play a game in your browser. Nothing is recorded: every seed is a new game, sampled live.

So where do we go from here?

As we’re now able to train 5v5 agents on modest hardware, in the next part we will work on fully porting 10 champions on modern League of Legends for patch 26.19 and training 5v5 agents on the full game.

Appendix

Observation and action space

Each of the ten heroes is its own agent. Every 133 ms of game time (four engine ticks, 7.5 decisions a second, the same rate OpenAI Five used) each hero receives one observation of 2,914 numbers and returns one action made of four choices. All ten share one network.

What a hero sees

The unit slots are fixed by class, nearest first within each class, and empty slots are zero:

One frame of reference. The purple team sees the map rotated half a turn, so “my base” is always in the same corner and “my top outer turret” is always the same slot. One network plays both sides without having to learn the map twice.

What it cannot see. There is no inventory: items are felt only through the stats they change. There are no buffs or debuffs beyond a single “I am disabled” flag. There is no history except the recent health changes; memory is the network’s job. And a hero sees only 16 enemy minions and 8 monsters at a time, the nearest ones.

For scale, OpenAI Five’s observation was about 16,000 numbers per hero. Ours is under a fifth of that, mostly because the game has fewer things in it: no items to choose, no courier, no wards, no runes.

What a hero can do

One action is four independent choices:

Aiming. Spells are aimed relative to the chosen target, not at a map coordinate. The nine bins for Q, W and R are offsets of 0, ±6, ±12, ±20 and ±35 degrees from the line to the target, which is enough to lead a moving enemy. E, usually a dash, gets eight compass directions relative to the target. Spells that lock onto a unit or cast on self use a single bin.

Targets. A target must be visible, targetable, an enemy or a monster, and within 1,500 units. That last rule is ours and exists for speed: without it the policy could order an attack on a turret across the map every decision, and the pathfinder re-planned the whole route each time, which made the simulator 20 to 28 times slower. To reach a far target the agent has to walk there.

The mask. A spell is legal only if it is learned, off cooldown, affordable and the hero is not stunned or silenced. The heads are sampled independently, so an illegal pairing (say, a unit-targeted spell with a turret as the target) falls back to the nearest legal target, or to doing nothing.

What the agent does not control

So these agents decide where to walk, what to hit, when to cast and when to Recall. Everything about builds is decided for them. OpenAI Five made the same split: item purchases, ability builds and courier control were scripted there too.

Two rules that exist only in training

Both were added to get Recall learned, and neither is in the real game.

How it changed

Every step was made on the live network without retraining from scratch. New inputs and the new action were added with zero weights, so the network’s behaviour was identical the moment before and after each change. The surgery appendix has the details.

Reward weights

These are the weights the final checkpoints were trained with, next to OpenAI Five’s for the nearest equivalent.

Individual rewards

Team rewards

Shared equally by all five heroes of the team.

What happens to the numbers afterwards

The two signs

Almost every individual weight is copied from Five. Two were not, and they point the other way.

Last hits. A last hit already pays twice, through the gold and the experience it gives. Five’s table has last hit at minus 0.16, to take some of that back. Ours has it at plus 0.16, on top. That is the “paid three times over” from the farming section, and with hindsight it is the first thing to change.

Kills. Five takes back 0.6 on a kill for the same reason: the gold and experience are already counted. We add 1.0, and 0.5 for each assist.

No time decay. A minion at minute 50 is worth exactly what it was at minute 5, while the win is worth 10 and may be half an hour away. With a discount half-life of 92 seconds, a win 30 minutes off is discounted to almost nothing today, and the minion in front of you pays now. The agents farm.

Network architecture

The network is a small copy of OpenAI Five’s design: encode every unit with the same small network, pool them, add the global state, run an LSTM, and pick a target by attention over the units. It has 8.07 million parameters. All ten heroes run the same weights, each on its own observation, with its own memory.

Units as a set. Every unit, whether a hero, a minion or a turret, goes through the same encoder, and each class is reduced to one vector by taking the maximum. The network therefore does not care what order minions arrive in or how many there are.

Choosing a target. The target head does not output “slot 17”. It outputs a query, and each unit’s score is how well its own encoding matches that query. The same encodings that describe the units to the network are the keys it picks from, so a unit that looks like a good target scores well wherever it sits in the list. This is the part taken most directly from Five.

Memory. The only memory is the LSTM state, carried from decision to decision and reset when a game resets. Policy and value have separate LSTMs and share only the encoders.

No communication. Nothing passes between the five heroes of a team. Each acts on its own observation, which includes its allies’ positions, health, cooldowns and targets. Whatever teamwork there is comes from that and from the reward being partly shared.

The value function sees what the policy sees. It is not given the enemy’s hidden information or the whole team’s observations, which is one of the simpler things we could change.

How it is trained

The 18-minute episode from a mid-game start is the setting the rest of the post argues with: no agent in this lineage ever played a whole game from minute zero during training.

Next to OpenAI Five

How the ladder was built

Every rating in this post comes from one rule: a game is won by destroying the nexus, and nothing else counts.

The games

Balancing sides and champion sets inside every pairing matters because both are worth a lot. The set with Caitlyn, Warwick, Nunu, Master Yi and Ryze wins 60% of games between equal policies. An unbalanced schedule would have rated the champions, not the agents.

The rating

We fit a Bradley–Terry model, which is the model behind Elo: each entrant has one number, and the chance that A beats B depends only on the difference between their numbers. A gap of 400 points means ten-to-one odds.

How to read the numbers

Two engines, one ladder

The bots exist only in the C# server, so those 640 games were played there, with our policy driving one team through the server’s own interface and the bots the other. That only works if a checkpoint is the same player on both engines. We checked with one pairing played on both:

What it cannot tell you

Roster Selection

Brokenwings isn’t a perfect implementation of the game: some champions are buggy or incomplete. Before porting anything we ran a stability test on the real server, 64 bot-only 5v5 games at once with forced fights and a 45-minute cap, three times over. A champion made the pool if its scripts ran a whole game without an exception and it had a behaviour tree, sourced from Riot’s original season 1 Lua files, that actually acts.

Because every champion in the pool runs as a bot, all ten have the full intermediate behaviour tree, which is what made the bot benchmark possible. Most League players know these bots from learning the game, and virtually all experienced players beat them consistently.

Ezreal and Garen were already ported from the 1v1 phase. The other eight were picked in pairs by kit, one pair per porting lane, and the ten were then split into two teams at random:

The split turned out to be lopsided. Playing random legal actions for about 1,400 games, the Caitlyn side won about 83% of the time on either side of the map, which is why the rosters swap sides every 1,200 rollouts in training. Not all of it was the champions, though:

Training Log

Training used the same model weights from the start until the last checkpoint. The model architecture was extended to provide the agents with additional inputs relating to items and allowing them to recall.

Neural network surgery used zero-padding to extend observations and action heads using the same method from OpenAI Five.