Detecting the Debugger on OS X

Jan 30, 2016

How can your program detect if it’s being attached to a debugger? One approach is detecting if a mach exception handler is configured. Another more well known approach is detecting the usage of ptrace. I will discuss the merits and disadvantages of both of these.

What does being Attached mean?

This question is easily neglected, but is necessary to answer so we understand what we are and aren’t detecting.

For our purposes, an inferior (the process being debugged) is attached to a debugger if the debugger catches unhandled exceptions that occur inside the inferior.

For example, when an inferior crashes or hits a breakpoint, a typical debugger may catch the exception and handle it by displaying the inferior’s state to the user while keeping the process suspended.

On the other hand, an example that falls short of attaching is reading/writing to a process’ virtual memory, which can even include overwriting code. We will not be discussing approaches that detect this type of behavior.

Mach Exception Handlers

Now that we settled for a definition, here is code that allows a program to detect if it is being attached via a mach exception handler¹:

#include <mach/task.h>
#include <mach/mach_init.h>
#include <stdbool.h>

static bool amIAnInferior(void)
{
	mach_msg_type_number_t count = 0;
	exception_mask_t masks[EXC_TYPES_COUNT];
	mach_port_t ports[EXC_TYPES_COUNT];
	exception_behavior_t behaviors[EXC_TYPES_COUNT];
	thread_state_flavor_t flavors[EXC_TYPES_COUNT];
	exception_mask_t mask = EXC_MASK_ALL & ~(EXC_MASK_RESOURCE | EXC_MASK_GUARD);

	kern_return_t result = task_get_exception_ports(mach_task_self(), mask, masks, &count, ports, behaviors, flavors);
	if (result == KERN_SUCCESS)
	{
		for (mach_msg_type_number_t portIndex = 0; portIndex < count; portIndex++)
		{
			if (MACH_PORT_VALID(ports[portIndex]))
			{
				return true;
			}
		}
	}
	return false;
}

We iterate through all ports an exception may use (EXC_BREAKPOINT, EXC_BAD_ACCESS, etc) and see if any of the ports are valid. If any of them are, then we know we are attached through a mach exception handler.

(You may wonder why the EXC_MASK_RESOURCE and EXC_MASK_GUARD exceptions are ignored; they are not supported)

Ptrace

ptrace is a system call found in OS X and other Unix-like systems that provides debugging capabilities. A common misconception on OS X is that a debugger needs to use ptrace for attaching to an inferior, which leads people to solely rely on detection code as shown in Apple’s Technical Q&A QA1361.

Such code is ineffective against debuggers that choose not to use ptrace though. Full-featured debuggers like LLDB and GDB tend to make use of ptrace, but most of the legwork needed for writing a capable debugger does not use this system call. Bit Slicer is just one example of a debugger that can set up exception handling for breakpoints without being caught by ptrace detection.

Some even believe ptrace is useless on our platform. Why would debuggers on OS X use ptrace then if that were not the case? Two reasons I can think of are²:

1. PT_TRACE_ME stops the inferior before executing its first instruction when a new process is spawned.
2. PT_SIGEXC allows all BSD signals to be received as mach exceptions.

Useful, but otherwise ptrace is not very interesting. Still, detecting if a debugger is attached via ptrace is not a bad idea because a debugger can use ptrace to trace execution without setting up a mach exception handler. However, detecting the usage of ptrace alone does not provide much coverage.

Anti-Debugging

Until now, I have avoided discussing anti-debugging measures that prevent debuggers from attaching. Although several measures exist, I am interested in discussing just two of them for the sake of our discussion:

1. Calling ptrace(PT_DENY_ATTACH, 0, NULL, 0). This denies any debugger that uses ptrace for attaching, but it won’t catch debuggers that don’t use ptrace.
2. Checking if a debugger has attached via amIAnInferior() and responding accordingly. The downside is it may not catch ptrace tracing or single stepping. This approach may also require polling.

Note the user can always break and return from your ptrace call, no-op your detection code in the executable, in memory, or wherever else. No program is truly safe from being debugged; we can only inconvenience the user’s efforts.

Conclusion

To sum up:

• A debugger attaches to another program if the debugger attempts to catch unhandled exceptions that occur within the inferior. If a debugger can do work without attaching to a process, then the debugger is not easily detectable.
• Detecting if a debugger is attached involves inspecting the mach exception ports or by inspecting if the ptrace flag has been set.
• ptrace is not required for writing an capable debugger, and shouldn’t be used alone for detecting if a debugger is attached.
• Both usage of PT_DENY_ATTACH and amIAnInferior() can be used for preventing debugger attachments.