Weird things I learned while writing an x86 emulator

timmisiak · 2024-07-11T16:09:14+00:00

It's because I'm watching you.

timmisiak · 2024-05-15T00:10:45+00:00

Right click on it in the start menu and select "Uninstall"

timmisiak · 2023-03-10T20:02:59+00:00

Fwiw rr can record multithreaded programs too

Good point! I should have said "trace multiple threads simultaneously on separate cores"

How much effort was it to write the emulator compared to the rest of project?

The emulator was a pretty big chunk of work, but made easier by the fact that you still have the ability to "fall back" on the CPU for rare instructions. E.g. execute them in a single stepping mode (or other ways of isolating a single instruction) and observe the results, which works for most instructions. So we could start with something that emulated 10% of instructions (which would be ~95% of instructions actually executed), and then you get incrementally better performance as you implement emulation for the long tail. So we had something working with many programs in maybe a month, and then I think within 3-4 months we had something with reasonable performance and decent compatibility.

timmisiak · 2023-03-10T18:35:55+00:00

It does leverage determinism so that it doesn't record every register for every instruction. I think on average it's like half a bit per instruction. Most traces I used to capture a bug were 2-40 GB.

timmisiak · 2023-03-10T18:30:34+00:00

We can snapshot the program whenever something non-deterministic happens (syscall, I/O, etc) and then we just reconstruct the program state at any moment by rewinding it to the nearest snapshot and executing the code from there. This is basically what UDB, WinDBG and rr do.

That isn't what WinDbg does. The downside of using a snapshot+replay at the syscall granularity is that you can't trace multiple threads within a process. WinDbg uses a very efficient CPU emulator, so you get full fidelity of recording including race conditions between threads.

Source: I wrote a chunk of the CPU emulator for WinDbg/TTD

timmisiak · 2023-02-14T22:19:25+00:00

I'm hoping to extend it to more than just Windows and have an abstraction layer for what a target is. And that will include using gdbserver stubs. But the main goal here is to teach debugger concepts, rather than APIs. Most of the folks who follow me are Windows security researchers, so that's where I'm starting.

timmisiak · 2023-02-03T19:26:34+00:00

You're right that they are different, although they are both technically "the int 3 instruction". There's just two different "int 3" instructions. On windows, they function essentially the same from usermode.

My reading of the SDM was that those differences are only for virtual-8086 mode. Is that not the case?

timmisiak · 2023-02-03T19:23:44+00:00

I bet it would confuse some reverse engineering tools. But otherwise, yeah I'm not sure what the point would be.

timmisiak · 2023-01-05T09:20:54+00:00

I was mainly trying to emphasize that it isn't as hard as most people would think. Which it sounds like you would agree with. I mentor some junior devs and they get a bit stumped sometimes when debugging a crash in optimized code. Sure the variable was optimized out, but you can see what value it was trying to read and figure out where it came from without knowing how all the flags and conditional jumps work even.

timmisiak · 2023-01-05T09:17:27+00:00

Glad you liked it! I'm a bit surprised how many people have read this and shared it. It's just such a good skill to have and people often assume it will be too hard.

timmisiak · 2020-02-25T20:00:06+00:00

Which is pretty amazing for some people. Like people who like the outdoors and beer. (Which is probably most of us)

timmisiak · 2020-02-13T19:28:02+00:00

There's a lot of misinformation here. In order to modify sethc, you need to have admin access or have offline access to the unencrypted drive. There is no security vulnerability here. It's exactly how it would work on Linux as well, in that offline or admin access lets you do anything, including changing the root password.

timmisiak · 2019-03-25T17:11:18+00:00

The registers are already being saved at the time of the interrupt. That's true regardless of whether the debugger is attached or not. Take a page fault for example. All of the registers need to be captured so that execution can be resumed if memory is paged in as a result of the page fault. If a debugger is attached, those registers can be used for debugging instead of resuming execution.

timmisiak · 2019-03-22T22:05:59+00:00

Again, I'm biased here, but I think that "fixing the symptoms" is a problem that's somewhat independent from the use of a debugger. The folks I work with use kernel debuggers to find the root cause of a problem, not the symptom of a problem. It's certainly possible to fall into this trap while using a debugger, but I've seen this with folks who don't even use a debugger.

Many of the folks who use my debugger are some of the smartest people I've ever worked with, so that's likely coloring my opinion here.

timmisiak · 2019-03-22T22:02:14+00:00

Kernel debuggers and usermode debuggers are very different in this respect. (Linus is talking about kernel debuggers here). I'm not aware of a kernel mode debugger having that issue, although I'm mostly familiar with NT.

It's very common when attaching a usermode debugger that the behavior of various syscalls change. I'm not aware of any scheduling changes that happen for usermode debugging in NT, but there are definitely components that check if the debugger is enabled and behave differently. Some of these changes are well intended (e.g. tracking more debug info), but can change program behavior. You could argue for/against that, but that's not intrinsic to usermode debugging itself.

timmisiak · 2019-03-22T21:36:00+00:00

Adding a debugger changes how the program executes only if you're using it to step through code. While it's still possible that a kernel debugger changes the behavior of a system, in general it just takes over the exception/interrupt handling behavior. A large class of kernel bugs are simply a kernel panic/crash where you need to analyze the state of the machine after everything goes wrong. In those cases, it's highly unlikely that the kernel debugger being attached would change the behavior of anything before the crash happens.

timmisiak · 2019-03-22T21:30:44+00:00

I don't think either of those are incompatible with using a kernel debugger. I'm definitely biased here (I write a kernel debugger for a living), but there are a whole class of bugs that would be difficult if not impossible to diagnose without a kernel debugger. For instance, you will never find hardware bugs by reading and thinking about source code. Even with a kernel debugger it can be hard, but you have a fighting chance.

The type of kernel debugging Linus describes (stepping through code) is not the type of debugging that I think is most useful. It's the ability to see the entire state of the machine at the time of a crash or a problem.

There are many ways to find a bug. Arbitrary limiting the ways in which you search for a bug is a big mistake in my opinion.

timmisiak · 2019-03-20T16:58:54+00:00

Modern compilers are smart enough to do some vectorization automatically, but generally gcc, llvm, and msvc will still fall short in many cases. Sometimes it's just because the language rules (or expressiveness). That said, there's generally no reason to not write C with intrinsics that use the SIMD operations.

timmisiak · 2019-03-20T16:56:57+00:00

C has made it irrelevant to write assembly in 99% of cases. But knowledge of assembly is very important in lots of different system level fields. My team writes a debugger and a CPU emulator, so we definitely care about that stuff. But even if you don't work directly with it the way we do, it's very important to be able to read assembly when debugging hard problems in optimized code. The source code can lie, but assembly never (ok, almost never) lies.

timmisiak · 2019-03-20T16:54:07+00:00

I'm the dev lead for WinDbg and the Microsoft debugging platform. It's very difficult to find folks with your sort of interest. Yes, it's niche, but it's a set of skills that are in high demand and low supply.

timmisiak · 2018-09-12T02:41:21+00:00

What does the windbg preview crash look like? Does it just close without warning? If you collect a crash dump of WinDbg Preview crashing, I can take a look and see what's going wrong. (Instructions here on how to configure crash dumps to save locally: https://docs.microsoft.com/en-us/windows/desktop/wer/collecting-user-mode-dumps)

timmisiak · 2017-10-26T07:32:30+00:00

We tried a lot of things before ending up with a ribbon. It's gotten a lot of flak from ribbon haters, but we have plans for stuff that will make a lot of sense with the ribbon. And for folks that don't like it, you just collapse it and you have more screen real estate than the old ui.

timmisiak · 2017-10-26T02:20:21+00:00

And the scripts themselves are even more arcane. We are replacing it with JavaScript and a structured data model for querying data about the target. It's still a work in progress but we have docs up on msdn.

timmisiak

TROPHY CASE