How can your program detect if it’s being attached to a debugger? One approach is detecting if a mach exception handler is configured. Another more well known approach is detecting the usage of
ptrace. I will discuss the merits and disadvantages of both of these.
What does being Attached mean?
This question is easily neglected, but is necessary to answer so we understand what we are and aren’t detecting.
For our purposes, an inferior (the process being debugged) is attached to a debugger if the debugger catches unhandled exceptions that occur inside the inferior.
For example, when an inferior crashes or hits a breakpoint, a typical debugger may catch the exception and handle it by displaying the inferior’s state to the user while keeping the process suspended.
On the other hand, an example that falls short of attaching is reading/writing to a process’ virtual memory, which can even include overwriting code. We will not be discussing approaches that detect this type of behavior.
Mach Exception Handlers
Now that we settled for a definition, here is code that allows a program to detect if it is being attached via a mach exception handler1:
We iterate through all ports an exception may use (
EXC_BAD_ACCESS, etc) and see if any of the ports are valid. If any of them are, then we know we are attached through a mach exception handler.
(You may wonder why the
EXC_MASK_GUARD exceptions are ignored; they are not supported)
ptrace is a system call found in OS X and other Unix-like systems that provides debugging capabilities. A common misconception on OS X is that a debugger needs to use
ptrace for attaching to an inferior, which leads people to solely rely on detection code as shown in Apple’s Technical Q&A QA1361.
Such code is ineffective against debuggers that choose not to use
ptrace though. Full-featured debuggers like LLDB and GDB tend to make use of
ptrace, but most of the legwork needed for writing a capable debugger does not use this system call. Bit Slicer is just one example of a debugger that can set up exception handling for breakpoints without being caught by
PT_TRACE_MEstops the inferior before executing its first instruction when a new process is spawned.
PT_SIGEXCallows all BSD signals to be received as mach exceptions.
Useful, but otherwise
ptrace is not very interesting. Still, detecting if a debugger is attached via
ptrace is not a bad idea because a debugger can use
ptrace to trace execution without setting up a mach exception handler. However, detecting the usage of
ptrace alone does not provide much coverage.
Until now, I have avoided discussing anti-debugging measures that prevent debuggers from attaching. Although several measures exist, I am interested in discussing just two of them for the sake of our discussion:
ptrace(PT_DENY_ATTACH, 0, NULL, 0). This denies any debugger that uses
ptracefor attaching, but it won’t catch debuggers that don’t use
- Checking if a debugger has attached via
amIAnInferior()and responding accordingly. The downside is it may not catch
ptracetracing or single stepping. This approach may also require polling.
Note the user can always break and return from your
ptrace call, no-op your detection code in the executable, in memory, or wherever else. No program is truly safe from being debugged; we can only inconvenience the user’s efforts.
To sum up:
- A debugger attaches to another program if the debugger attempts to catch unhandled exceptions that occur within the inferior. If a debugger can do work without attaching to a process, then the debugger is not easily detectable.
- Detecting if a debugger is attached involves inspecting the mach exception ports or by inspecting if the
ptraceflag has been set.
ptraceis not required for writing an capable debugger, and shouldn’t be used alone for detecting if a debugger is attached.
- Both usage of
amIAnInferior()can be used for preventing debugger attachments.