Buffer Overflow Isn't as Scary as You Think: From "an Overflowing Cup" to Getting a Shell

This one is a bonus spin-off from my last post, the MBPTL walkthrough. In that write-up the final stage (Phase 7, the 31337 binary) ended with me just showing the result — "send a bunch of A's plus three addresses and you get a shell" — without ever explaining why. So this post is dedicated to taking that string of what looks like black magic and pulling it apart in the plainest terms possible. No pwn background required. By the end you'll realize it's really not as scary as it looks.

One sentence first: this bug is really just "writing too much, until it overflows"

You've definitely hit a web form that says "20 characters max." That limit isn't there to annoy you — it's because the program has set aside a chunk of space that fits exactly 20 slots to hold your text. If the program forgets to check the length and you cram 200 characters in there, what happens?

The answer: the extra characters don't just vanish into thin air. They overflow into the memory next door and overwrite whatever was already sitting there. That's a buffer overflow — the name sounds hardcore, but the concept is just "fill the cup too full and the water spills onto the table."

The catch is: the water that spills over happens to flood something very important.

Diagram 1: normal input vs. cramming it full

First, let's see how the program lays things out in memory. It sets aside a row of slots (called the buffer) to hold your input, and directly "above" that row of slots is hidden something called the "return address" — a note recording where the program should go next once this chunk of work is done.

On the left is the normal case: you type "Kevin," fill only a few slots, and the "return address" above sits safe and sound. On the right is the attack: you type 140 A's, fill all 128 slots and still have leftovers, and those extra A's flood all the way up and clobber the "return address" slot too.

(One line of technical detail: this lab's program uses an old function called gets() to read input — notoriously dangerous because it doesn't check the length at all. However long you feed it, it takes it all. Modern code should never touch it, but it's perfect as a teaching example.)

Diagram 2: clobbering the "return address" is like grabbing the steering wheel

Just overwriting one slot of data — that doesn't sound like a big deal, right? The key is that this "return address" slot isn't ordinary data, it's the program's next instruction.

Every time the program finishes a chunk of work, it looks back at this slot and jumps to whatever address is written there to keep executing. Since we just overwrote it with an address of our own choosing, the program will obediently jump wherever we tell it to.

This step is the soul of the whole attack: upgrading from "I can overwrite one slot of memory" to "I can decide where the program jumps next." The steering wheel is yours.

Diagram 3: jump to where, exactly? Build a shell out of spare parts

The steering wheel is yours — but a new problem shows up: jump to where to actually seize control? What we want is a shell (a command line where we can run commands), but the program doesn't contain a ready-made chunk that says "please give me a shell."

The good news: the program has parts. This lab's binary happens to have two things lying around inside it — a function that can "run a command" (called system), and a piece of text that spells out /bin/sh (the name of the shell itself). What we need to do is assemble these two parts into "run /bin/sh." This technique is called ret2libc.

Assembly takes three steps. There's a rule you need to know first: on this kind of computer, to tell a function to "handle something," you first have to put that something's address into a register called rdi (think of it as the "function's inbox"). So the order is: first put /bin/sh in the inbox, then tell system to get to work.

What is that "① gadget"? It's a tiny fragment we dig out of the program's existing machine code, one that happens to do exactly "put a value into the inbox, then continue." We can't conjure code out of nowhere and write it in (I'll explain why later), but we can borrow the parts the program is already made of — that's the clever bit of this technique. At no point do we smuggle in a single line of our own code; we just rearrange someone else's building blocks.

And the bottom half of the diagram reveals what that mysterious payload really is: it's just "a pile of padding + three addresses" stacked in memory in order. The padding fills the hole up to exactly where it butts against the "return address," and then the three addresses, one after another, lead the program like a marionette, step by step, into system("/bin/sh").

Back to that "magic string you couldn't read" from the last post — line it up now and it's clear:

payload = b'A' * 136        # padding: fill the hole, butting right up against the return address
        + p64(gadget)       # ① clobber the return address → jump to the gadget
        + p64(binsh)        # ② the value the gadget drops into the inbox = "/bin/sh"
        + p64(system)       # ③ finally jump to system, get to work

That "136" isn't a random number, it's precisely calculated — the buffer is 128 slots, plus 8 more in between, so exactly after 136 slots comes the "return address." One slot too many or too few and it fails.

So why don't most modern programs get owned like this? Three locks

By now you might be thinking: this is so easy, isn't every program in the world doomed? Actually no — modern compilers long ago fitted three layers of defense that make every step above brutally hard. The reason this lab sails straight through is that it deliberately turns all three locks off for teaching purposes.

• The canary (Stack Canary): it quietly plants a random codeword right in front of the "return address." Before the program jumps back, it first checks whether the codeword is still intact — when you overflow, you're guaranteed to overwrite it, and the moment the codeword doesn't match, the program self-destructs on the spot, before you ever get a chance to hijack anything. The name comes from miners taking a canary down the mine: the bird dies first, and people know there's toxic gas.
• Address randomization (PIE / ASLR): every time the program runs, the parts get shuffled to random locations. Those three "addresses" in the payload above? You simply can't hardcode them — this run it's at one spot, next run it's somewhere else. (If you've ever played with ARM exploitation, this one won't be new to you; bypassing ASLR is bread and butter for the advanced challenges.)
• Non-executable stack (NX): early attacks just stuffed malicious code straight into the buffer and ran it. The NX lock decrees "this chunk of memory can only hold data, not run as code." This is exactly why we don't stuff in our own code, but instead go "borrow" the program's existing parts — the ret2libc technique was born precisely to get around NX.

Turn on any single one of these three locks and the difficulty jumps a whole tier. Real-world programs usually have all three on, which is why modern pwn experts spend most of their effort on "how to peel off these locks one by one" — and that's a much, much deeper rabbit hole.

Wrapping up

Stringing the whole thing together in one line:

The program forgot to check the length → I cram the buffer full, flood over, and clobber the "return address" → so now I can decide where the program jumps next → and using the program's own parts, I lead it step by step into a shell.

Look back at that b'A'*136 + three addresses from the last post, and it shouldn't be black magic anymore — it's just "fill the padding to exactly the right point, then use three addresses to lead the program by the nose."

If you're like me and used to skip right past anything labeled pwn, I hope this post left you feeling: it's tough, sure, but the skeleton is actually pretty easy to grasp. The genuinely hard part isn't this intro version — it's how you get around all three locks once they're on. But that one, let's save for the day you actually decide to jump into this rabbit hole.