How to reverse engineer a binary: a reproducible first workflow

The first goal of reverse engineering is to answer a precise question about a program. It is not to rename every function or recreate an entire source tree. A useful beginner question is: “Which inputs does this program accept, and where does it reject them?” That question gives you a path through strings, control flow, data types, and evidence.

Use a binary you compiled yourself for this exercise. You know its provenance, can run it safely, and can compare your conclusions with source after the investigation. Reserve unfamiliar executable samples for an isolated analysis environment. The workflow below uses Ghidra concepts retained by the Opcodex project; check the installed application’s help when menu labels differ.

1. Create an understandable target

Save this small program as ticket.c. It accepts a deliberately simple command-line value and prints a result. It is a training parser, not an authentication design.

#include <stdio.h>
#include <string.h>

int main(int argc, char **argv) {
    if (argc != 2) {
        puts("usage: ticket VALUE");
        return 2;
    }
    size_t n = strlen(argv[1]);
    if (n != 6) {
        puts("wrong length");
        return 1;
    }
    if (argv[1][0] != 'T' || argv[1][5] != '7') {
        puts("wrong marker");
        return 1;
    }
    puts("accepted");
    return 0;
}

On a system with a compatible C compiler:

cc -O0 -g -o ticket ticket.c
./ticket
./ticket T12347
./ticket A12347

The expected observations are a usage message, acceptance, and rejection. Record exit statuses as well as printed text. Avoid optimized compilation for your first pass; then make an optimized copy and compare how the same behavior appears after transformations.

2. Preserve identity before interpretation

Create a working directory containing the original binary, a notes file, and a place for exports. Record its cryptographic hash. In PowerShell, for a Windows build of the exercise:

Get-FileHash -Algorithm SHA256 -LiteralPath .\ticket.exe

Write down your compiler version, build flags, operating system, and the application version used for analysis. A hash identifies the exact bytes. The filename alone does not: rebuilding ticket.exe can produce a different artifact while leaving the name unchanged.

Keep your untouched input separate from patched or transformed copies. If you later experiment with patching a branch, the original evidence should remain available. A beginner project benefits from the same discipline as a larger investigation because it lets you explain unexpected differences.

3. Import and check assumptions

Create a local project and import the executable. Review the detected format and language before continuing. For a recognized executable, use the loader’s metadata rather than selecting a processor from a guess. A raw binary needs additional knowledge, including its architecture and memory layout.

The official Ghidra introduction covers import, analysis, and the Code Browser’s principal windows. Open the imported program and run automatic analysis with the defaults for this simple target. Save the program when it finishes. Record warnings instead of assuming that every discovered function boundary is final.

Before reading pseudocode, confirm that the memory map and entry address are plausible for your file. Wrong architecture or loading assumptions can generate impressive-looking nonsense. Correct the import first; do not fix that problem by renaming the resulting fragments.

4. Start from an observable string

Find wrong length or accepted in the program’s defined strings or string search. Follow references to the code that uses it. You should reach the decision logic in or near the program’s main routine, although compiler and runtime scaffolding may surround it.

Look at the Listing and Decompiler together. The Listing tells you which instructions and addresses support the high-level presentation. The Decompiler provides a more compact hypothesis about behavior. Neither replaces the need to follow branches. Make a small table as you trace the function:

For this ASCII exercise, bytes and displayed characters coincide. Do not generalize that to arbitrary Unicode input or every string API.

5. Add names only when justified

Rename an unidentified length variable to input_length after establishing where it comes from. Rename a helper to check_ticket_marker only if it really represents that operation; this example may keep all logic in main. Prefer a neutral name such as candidate_input when intent remains uncertain.

Check parameter and return types against callers. A value that happens to be zero on one branch is not automatically a Boolean. An address used to read characters is stronger evidence for a string pointer than a default autogenerated type name. Document uncertain interpretations in comments so your future self does not confuse them with recovered source names.

6. Validate a behavioral model

Write a plain-language specification before looking back at the source: one command-line argument, length six, first byte T, last byte 7; return zero on acceptance and nonzero on the observed rejection paths. Test the middle characters, too. The source does not constrain them, so a model claiming a fixed password is wrong.

Keep a tiny regression checklist:

• No argument reaches the usage path.
• Five-byte and seven-byte inputs fail the length check.
• Changing either marker causes rejection.
• Changing only a middle ASCII byte preserves acceptance.
• Printed text and exit status agree with the recorded model.

These tests validate the model on selected inputs. They are not a proof about every possible input. That distinction matters when you move from a toy parser to a real file format.

7. Write a result someone can continue

Finish with the input hash, tool version, important addresses, recovered rule, supporting tests, and open questions. Include the compiler flags because they explain much of the difference between your debug and optimized builds. Keep source comparison as a final learning step rather than a shortcut around the exercise.

Next, improve the model with types and decompiler annotations. For unattended processing, see the headless documentation. Opcodex’s overview distinguishes the Ghidra engine foundation from planned extensions, so you can build a workflow around capabilities that are actually available.