Introduction & Core Concepts
Reverse Engineering is a foundational discipline in application security auditing, vulnerability research, and proprietary software analysis. When evaluating the security of a native binary (C/C++, Rust, Go, Swift, iOS/Android apps, or embedded IoT firmware), access to the original source code is rarely available.
Professional binary analysis combines two complementary methodologies:
- Static Analysis: Inspecting the disassembled and decompiled binary using Ghidra (the NSA's open-source framework).
- Dynamic Analysis (Hooking & Instrumentation): Intercepting and modifying live function calls in memory using Frida.
1. Target Program: C License Checker
We start with a compiled C binary that protects access to a sensitive feature using a serial key.
❌ Vulnerable C Source Code (auth_system.c)
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
// ❌ VULNERABILITY: The secret key is stored in plaintext in the .rodata section
// and the access logic relies solely on the integer return value of an isolated function.
int check_serial_key(const char *input_key) {
const char *master_key = "BMJ-SECRET-2026-X99";
if (strcmp(input_key, master_key) == 0) {
return 1; // Success
}
return 0; // Failure
}
int main(int argc, char **argv) {
if (argc < 2) {
printf("Usage: %s <serial_key>\n", argv[0]);
return 1;
}
printf("[*] Verifying serial number...\n");
if (check_serial_key(argv[1])) {
printf("[+] ACCESS GRANTED! Welcome to the admin terminal.\n");
} else {
printf("[-] ACCESS DENIED. Invalid key.\n");
}
return 0;
}
gcc -O2 auth_system.c -o auth_system
2. Step 1 — Static Analysis in Ghidra
- Import & Analyze: Open Ghidra, create a project, import
auth_system. Run Auto Analyze. - Locate Functions: In the Symbol Tree, expand
Functions, findmain. - Decompiler Output: Ghidra reconstructs near-C source code:
/* Pseudo-code generated by Ghidra Decompiler */
undefined8 main(int param_1, long param_2) {
int iVar1;
if (param_1 < 2) {
printf("Usage: %s <serial_key>\n", *(char **)param_2);
return 1;
}
puts("[*] Verifying serial number...");
// 👈 Ghidra reveals the secret key string and function call directly!
iVar1 = check_serial_key(*(char **)(param_2 + 8));
if (iVar1 == 1) {
puts("[+] ACCESS GRANTED! Welcome to the admin terminal.");
} else {
puts("[-] ACCESS DENIED. Invalid key.");
}
return 0;
}
The secret key "BMJ-SECRET-2026-X99" is immediately visible in the decompiler window!
3. Step 2 — Dynamic Hooking with Frida
Instead of brute-forcing the key, we write a Frida script that intercepts the call to check_serial_key at runtime and patches its return value to 1.
✅ Frida Instrumentation Script (bypass_hook.js)
console.log("[*] Frida binary audit script loaded!");
const targetFunctionPtr = Module.findExportByName(null, "check_serial_key");
if (targetFunctionPtr) {
console.log("[+] Found check_serial_key at: " + targetFunctionPtr);
Interceptor.attach(targetFunctionPtr, {
onEnter: function (args) {
const keyPassed = Memory.readUtf8String(args[0]);
console.log("[+] onEnter: User supplied key = '" + keyPassed + "'");
},
onLeave: function (retval) {
console.log("[!] onLeave: Original return value = " + retval.toInt32());
// Patch return value to 1 (success) regardless of input
retval.replace(ptr(1));
console.log("[+] Return value patched to 1 — bypass successful!");
}
});
} else {
console.log("[-] Function 'check_serial_key' not found.");
}
frida -l bypass_hook.js --no-pause -f ./auth_system -- "WRONG_KEY_ENTIRELY"
Console output:
[*] Verifying serial number...
[+] onEnter: User supplied key = 'WRONG_KEY_ENTIRELY'
[!] onLeave: Original return value = 0
[+] Return value patched to 1 — bypass successful!
[+] ACCESS GRANTED! Welcome to the admin terminal.
The access check is bypassed entirely in memory without modifying the file on disk.
4. Remediation & Native Code Hardening
#include <stdio.h>
#include <string.h>
#include <openssl/sha.h>
#include <sys/ptrace.h>
// 1. Anti-debugging detection (detects Frida / GDB attachment)
void check_anti_debug() {
if (ptrace(PTRACE_TRACEME, 0, 1, 0) < 0) {
printf("[ALERT] Debugger/Frida detected! Exiting.\n");
exit(1);
}
}
// 2. Constant-time hash comparison (prevents timing attacks and plaintext exposure)
int check_serial_secure(const char *input_key) {
const unsigned char expected_hash[SHA256_DIGEST_LENGTH] = { /* pre-computed SHA-256 bytes */ };
unsigned char user_hash[SHA256_DIGEST_LENGTH];
SHA256((const unsigned char*)input_key, strlen(input_key), user_hash);
return (CRYPTO_memcmp(user_hash, expected_hash, SHA256_DIGEST_LENGTH) == 0);
}
Key Takeaways
- Ghidra maps program structure and reveals plaintext secrets in
.rodata. - Frida validates vulnerabilities by patching memory at runtime without touching the binary on disk.
- Never store secrets in plaintext in native binaries — use salted hashes and anti-debug guards.