How do I debug an operating system

How does a debugger work?

I keep wondering how a debugger works. In particular, the one that can be attached to an already running executable. I understand that the compiler translates code into machine language, but then how does the debugger know what to append to?


The details of how a debugger works depends on what you are debugging and what operating system you are using. For native debugging on Windows, there are some details about MSDN: Win32 Debugging API.

The user tells the debugger by name or process ID which process to connect to. If it is a name, the debugger looks up the process ID and initiates the debug session through a system call. On Windows this would be DebugActiveProcess.

Once attached, the debugger enters an event loop, similar to any user interface. Instead of events coming from the window system, the operating system generates events based on what is happening in the process being debugged, such as an exception that has occurred. See WaitForDebugEvent.

The debugger can read and write the virtual memory of the target process and even adjust its register values ​​via APIs provided by the operating system. See the list of debugging functions for Windows.

The debugger can use information from symbol files to translate addresses into variable names and locations in the source code. The information about the symbol file is a separate API and not a central part of the operating system. In Windows, this is done using the Debug Interface Access SDK.

When debugging a managed environment (.NET, Java, etc.) the process is usually similar but the details are different because the virtual machine environment provides the debug API instead of the underlying operating system.

For me, it sounds like:

In the case of software breakpoints on x86, the debugger replaces the first byte of the instruction with (). This is done under Windows. If the CPU arrives at this instruction and executes it, the CPU generates a debug exception. The operating system receives this interrupt, detects that the process is being debugged, and notifies the debugger process that the breakpoint has been reached.

After the breakpoint has been reached and the process stopped, the debugger searches its list of breakpoints and replaces the byte with the original byte. The debugger sets, the trap flag in (by modifying) and continues the process. The trap flag causes the CPU to automatically generate a single-step exception () for the next command.

The next time the process to be debugged stops, the debugger again replaces the first byte of the breakpoint statement and the process continues.

I'm not sure that this is exactly how all debuggers implement this, but I've written a Win32 program that can debug itself using this mechanism. Completely useless, but educational.

On Linux, debugging a process begins with the ptrace (2) system call. This article has a great tutorial on implementing some simple debugging constructs.

If you are using a Windows operating system, "Debugging Applications for Microsoft .NET and Microsoft Windows" by John Robbins is an excellent resource:

(or even the older edition: "Debugging Applications")

The book contains a chapter on how a debugger works that contains code for some simple (but working) debuggers.

Since I'm not familiar with the details of Unix / Linux debugging, this stuff may not apply to other operating systems at all. But I would guess that as an introduction to a very complex topic, the concepts - if not the details and APIs - should be "ported" to most operating systems.

Another valuable resource for understanding debugging is the Intel CPU Manual (Intel® 64 and IA-32 Architectures Software Developer's Manual). In Volume 3A, Chapter 16, hardware support for debugging was introduced, e.g. B. special exceptions and hardware debugging registers. The following is from this chapter:

T (Trap) Flag, TSS - Generates a debug exception (#DB) when attempting to switch to a task that has the T flag set in its TSS.

I'm not sure whether Windows or Linux use this flag or not, but this chapter is very interesting to read.

Hope this helps someone.

I think here are two main questions to answer:

1. How does the debugger know that an exception has occurred?

When an exception occurs in a process that is being debugged, the operating system notifies the debugger before any user exception handlers defined in the target process are given a chance to respond to the exception. If the debugger does not handle this exception notification (first chance), the exception distribution sequence will continue and the target thread will be given the opportunity to handle the exception if it chooses. If the SEH exception is not being handled by the target process, another debug event, known as a second chance notification, is sent to the debugger to inform it that an unhandled exception has occurred in the target process. source

2. How does the debugger know how to stop at a breakpoint?

The simplified answer is, if you put a breakpoint in the program, the debugger will replace your code at that point with an int3 instruction which is a software interrupt. As a result, the program stops and the debugger is called.

I understand that when I compile an application or DLL file, everything that is compiled contains symbols that represent the functions and variables.

If you're on a debug build, these icons are far more detailed than they are on a release build, so the debugger can give you more information. Attaching the debugger to a process checks which functions are currently being accessed and resolves all available debugging symbols from here (knowing what the internals of the compiled file are, it can determine what may be in memory, with contents of ints, floats, strings, etc.). As the first poster said, this information and how these icons work is highly dependent on the environment and language.

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from.

By continuing, you consent to our use of cookies and other tracking technologies and affirm you're at least 16 years old or have consent from a parent or guardian.

You can read details in our Cookie policy and Privacy policy.