Anti-Debug: Assembly instructions
- 1. INT 3
- 2. INT 2D
- 3. ICE
- 4. Stack Segment Register
- 5. Instruction Counting
- 6. POPF and Trap Flag
- 7. Instruction Prefixes
The following techniques are intended to detect a debugger presence based on how debuggers behave when the CPU executes a certain instruction.
Instruction INT3 is an interruption which is used as a software breakpoint. Without a debugger present, after getting to the INT3 instruction, the exception EXCEPTION_BREAKPOINT (0x80000003) is generated and an exception handler will be called. If the debugger is present, the control won’t be given to the exception handler.
Besides the short form of INT3 instruction (0xCC opcode), there is also a long form of this instruction: CD 03 opcode.
When the exception EXCEPTION_BREAKPOINT occurs, the Windows decrements EIP register to the assumed location of the 0xCC opcode and pass the control to the exception handler. In the case of the long form of the INT3 instruction, EIP will point to the middle of the instruction (i.e. to 0x03 byte). Therefore, EIP should be edited in the exception handler if we want to continue execution after the INT3 instruction (otherwise we’ll most likely get an EXCEPTION_ACCESS_VIOLATION exception). If not, we can neglect the instruction pointer modification.
Just like in the case of INT3 instruction when the instruction INT2D is executed, the exception EXCEPTION_BREAKPOINT is raised as well. But with INT2D, Windows uses the EIP register as an exception address and then increments the EIP register value. Windows also examines the value of the EAX register while INT2D is executed. If it’s 1, 3 or 4 on all versions of Windows, or 5 on Vista+, the exception address will be increased by one.
This instruction can cause problems for some debuggers because after the EIP incrimination, the byte which follows the INT2D instruction will be skipped and the execution might continue from the damaged instruction.
In the example, we put one-byte NOP instruction after INT2D to skip it in any case. If the program is executed without a debugger, the control will be passed to the exception handler.
“ICE” is one of Intel’s undocumented instructions. Its opcode is 0xF1. It can be used to detect if the program is traced.
If ICE instruction is executed, the EXCEPTION_SINGLE_STEP (0x80000004) exception will be raised.
However, if the program has been already traced, the debugger will consider this exception as the normal exception generated by executing the instruction with the SingleStep bit set in the Flags registers. Therefore, under a debugger, the exception handler won’t be called and execution will continue after the ICE instruction.
This is a trick that can be used to detect if the program is being traced. The trick consists of tracing over the following sequence of assembly instructions:
After single-stepping in a debugger through this code, the Trap Flag will be set. Usually it’s not visible as debuggers clear the Trap Flag after each debugger event is delivered. However, if we previously save EFLAGS to the stack, we’ll be able to check whether the Trap Flag is set.
This technique abuses how some debuggers handle EXCEPTION_SINGLE_STEP exceptions.
The idea of this trick is to set hardware breakpoints to each instruction in some predefined sequence (e.g. sequence of NOPs). Execution of the instruction with a hardware breakpoint on it raises the EXCEPTION_SINGLE_STEP exception which can be caught by a vectored exception handler. In the exception handler, we increment a register which plays the role of instruction counter (EAX in our case) and the instruction pointer EIP to pass the control to the next instruction in the sequence. Therefore, each time the control is passed to the next instruction in our sequence, the exception is raised and the counter is incremented. After the sequence is finished, we check the counter and if it is not equal to the length of our sequence, we consider it as if the program is being debugged.
This is another trick that can indicate whether a program is being traced.
There is a Trap Flag in the Flags register. When the Trap Flag is set, the exception SINGLE_STEP is raised. However, if we traced the code, the Trap Flag will be cleared by a debugger so we won’t see the exception.
This trick works only in some debuggers. It abuses the way how these debuggers handle instruction prefixes.
If we execute the following code in OllyDbg, after stepping to the first byte F3, we’ll immediately get to the end of try block. The debugger just skips the prefix and gives the control to the INT1 instruction.
If we run the same code without a debugger, an exception will be raised and we’ll get to except block.
- During debugging:
- The best way to mitigate all the following checks is to patch them with NOP instructions.
- Regarding anti-tracing techniques: instead of patching the code, we can simply set a breakpoint in the code which follows the check and run the program till this breakpoint.
- For anti-anti-debug tool development: No mitigation.