ExploitGym Explained: 869 Real Vulnerabilities and AI That Actually Writes Exploits

While recently chatting about the security capabilities of AI models, I came across a name: ExploitGym.

My first reaction was: what's this, another new lab platform? Something like Hack The Box or VulnHub, or maybe some kind of CTF for AI to play?

After digging in, I realized it's actually pretty different.

ExploitGym doesn't ask an AI to find a SQL injection, and it doesn't hand the model a few deliberately crafted pwn challenges to solve. It takes real software vulnerabilities, first gives the model a PoV that already crashes the program, and then requires the model to develop that crash all the way into a complete exploit, ultimately capturing a secret flag inside the target environment.

That difference matters.

In the past, a lot of AI security benchmarks tested this:

Find the vulnerability
→ Produce an input that crashes it
→ Task complete

ExploitGym tests this:

A known vulnerability causes a crash
→ Understand how the vulnerability works
→ Build a read/write primitive
→ Bypass or handle memory layout
→ Complete the exploit chain
→ Achieve arbitrary code execution
→ Read the secret flag

It's no longer just "can you find a bug." It's asking:

Can the AI actually turn a known vulnerability into a real attack?

The short version first

Here's what stands out about ExploitGym right now:

• The public task set contains 869 challenges
• The challenges come from real userspace, Chrome V8, and Linux kernel vulnerabilities
• Each challenge provides source code, build instructions, a PoV, and an isolated execution environment
• The goal is to achieve code execution and read a secret flag
• Humans can technically play, but there's no HTB-style operating interface
• Officially there's no published human baseline or human leaderboard yet

Both the project and the paper are public:

Project: sunblaze-ucb/exploitgym
Official overview: ExploitGym
Paper: ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

How are the 869 challenges made up?

The current public version breaks down into three categories:

Userspace

This section is mostly C/C++ projects sourced from OSS-Fuzz, OSV, and CyberGym. The kinds of things you might run into include:

• Heap overflow
• Use-after-free
• Out-of-bounds read/write
• Integer overflow
• Stack corruption
• Format string
• All sorts of weird parser and codec issues

Some of the targets even come from large real-world projects like FFmpeg and OpenSSL.

Chrome V8

This section dives straight into browser engine exploitation:

• JIT compiler bugs
• Type confusion
• Out-of-bounds access
• Fake objects
• Arbitrary read/write
• Pointer leaks
• Sandbox escape

This is well beyond the difficulty of your typical CTF pwn challenge.

Linux Kernel

Kernel challenges run inside a virtual machine, and the goal is usually to go from low privilege all the way to full privilege escalation.

You may need to deal with:

• Kernel heap
• Race conditions
• UAF
• modprobe_path
• Kernel ROP
• SMEP/SMAP
• KASLR
• Various kernel hardening measures

Put these three categories together and you start to understand why "has any human beaten all of them" is a slightly terrifying question. Each of these three areas is a field you could spend years studying on its own.

Why do some places say 898 challenges?

The early data snapshot used in the ExploitGym paper had 898 challenges, but by the time it was officially released, it had become 869 challenges.

The reason is that the research team later re-examined the challenges and removed 29 that were judged to be:

• Not reliably exploitable in the specified environment
• Problematic in their PoV or environment
• Not suitable for inclusion in the official public benchmark

So neither number is arbitrary:

Early paper snapshot: 898
Official public task set: 869

The official challenge list lives at:

data/task_ids/v1.txt

The project also provides one with only 20 challenges:

data/task_ids/sample.txt

If you want to set it up and study it yourself, you don't need to pull down all 869 target images right from the start.

Can humans play?

Yes, but it isn't designed for the average player.

ExploitGym is open source. The target images, controller, agent container, and scoring tools can all be run yourself.

Humans can of course fire up a target and then use:

• GDB
• pwndbg
• Python
• pwntools
• nc
• QEMU
• Your own fuzzer or exploit script

to work through challenges by hand.

The basic setup flow looks roughly like this:

git clone https://github.com/sunblaze-ucb/exploitgym
cd exploitgym

uv sync --extra proxy
bash scripts/setup/setup_data.sh
bash scripts/setup/validate.sh

Then pull the sample challenges first:

uv run scripts/setup/pull_images.py data/task_ids/sample.txt
uv run scripts/setup/pre_run.py data/task_ids/sample.txt

But it doesn't have:

• A player-oriented web interface
• Difficulty tags or challenge category pages
• A hint system
• Writeup unlocks
• A human mode
• A human scoreboard

So it's more like:

An AI exploit lab that's open for humans to come in and study.

Rather than another HTB.

The most interesting part isn't the success rate — it's that the AI goes off script

ExploitGym has one design choice that I think is beautiful:

Getting the flag doesn't necessarily count as actually solving it.

Besides checking whether the agent obtained the flag, the system uses a judge to confirm whether it actually exploited the specific vulnerability the challenge intended. And it turns out that AI very often goes off and finds some other hole instead.

One set of results the team published is this:

In other words, GPT-5.5 grabbed the flag on 90 challenges without going through the vulnerability that was originally provided.

Some agents will:

• Discover a nearby code path with weaker validation
• Decide the original vulnerability is hard to exploit and just re-audit the source code from scratch
• Do dynamic fuzzing against the target
• Find an attack surface the challenge author never intended

From a benchmark's point of view, that counts as failing to complete the assigned task.

But from the point of view of a real red team or vulnerability research, this is actually even scarier.

It's not just following the challenge's instructions to solve it — it discovers:

You told me to hit this hole, but the one next to it is easier, so I'll go in through there first.

Who is ExploitGym for?

It's not a good choice for your first pwn lab. I'd recommend being comfortable with the following concepts:

• Stack/heap
• ELF
• GDB
• ROP
• ASLR
• Canary
• Use-after-free
• Arbitrary read/write

If you jump straight into ExploitGym, you'll probably just see a pile of massive source code and crash logs and start questioning your life choices.

I think a more sensible path is:

pwn.college
→ General CTF pwn challenges
→ Reproducing real CVEs
→ ExploitGym sample.txt
→ Pick a single userspace challenge to study
→ V8/Kernel

You don't need to have the mindset of "I'm going to beat all 869 challenges."

Just picking a single challenge and doing a full comparison:

• Analyze it manually yourself
• Let different agents run it
• Compare the exploit chains from both sides
• Toggle mitigations on and off
• Check whether the agent went through the expected vulnerability

is already a really interesting research topic in itself.

Closing thoughts

The most noteworthy thing about ExploitGym isn't how many points a particular model scored.

It's that the evaluation of AI security capabilities has already moved from:

Can you spot the vulnerability here?

all the way to:

Here's a real vulnerability and a crash.
Can you turn it into a complete exploit yourself?

We can't yet use a leaderboard to declare that AI has surpassed every human exploit researcher.

But one thing is already clear:

Automated exploit development is no longer just a "this might happen someday" fantasy.

It can already be tested, reproduced, and compared at scale — and we're even starting to see results like "the model didn't follow the challenge and found another hole on its own."

That's the part of ExploitGym that really sends a chill down your spine.

References

• ExploitGym official overview
• ExploitGym GitHub
• ExploitGym Changelog
• ExploitGym Sample Task List
• ExploitGym paper
• pwn.college