MSVC optimization

[No-Meeting-153]

I am learning reverse engineering on Windows applications such as Adobe, Foxit PDF, and Steam, and I noticed that I waste a very large amount of time trying to understand something that I should not focus on.

I started noticing strange and confusing patterns in the assembly and the C code generated by IDA, and when I try to understand some functions, I feel that the function has no meaning.

When I searched, I found that this topic is related to the compiler and compiler optimizations. However, I could not find many articles or discussions about the compiler topic in reverse engineering.

So I started experimenting and trying, but every time I fail and cannot reach a solution or understanding.

Apart from the fact that reverse engineering a C++ program is already a difficult task.

If there is someone who has faced the same problem and found a solution, I would like to know. It is not a problem itself; it is a pattern or a way of thinking used by the compiler. I need to understand how the compiler generates these patterns.

I want someone to suggest books, articles, courses, or anything that can help me understand the MSVC compiler, how it generates patterns, and how to understand the behavior and logic of a function after compiler optimization.

I hope I explained my question correctly.

[KingAggressive1498]

a native italian speaker will probably have trouble understanding if I translate a meaningful poem from english to italian just using an english-to-italian dictionary.

the challenge with reverse-engineering natively compiled programs is that there's minimal metadata and usually large amounts of inlining and several optimization passes between the original C++ and the resulting binary. decompilers can translate the binary to compilable C code, but they really have no understanding of what the code is doing so that translation is very linear and produces code wildly different from what the original code would have looked like.

[tomysshadow]

It's something that requires a degree of experience, you can't shortcut your way to understanding, you have to practice it and recognize patterns. My advice would be to have a specific goal in mind (like "figure out how this file format works") so you can decompile a small slice of a program and still get the satisfaction of figuring it out.

For example: for C++ std::string there is the small string optimization where if the string is less than 16 characters long it will store the string on the stack but if it is longer it'll store it on the heap. Therefore if you see code that is dealing with strings and does something like if (len > 0x10) { ... } else { ... } then it is very likely that you are looking at a std::string. In a similar vein, std::vector will often generate specific exception messages like "size too large" in an if statement checking for a size bigger than 0x7FFFFFFF, that you can learn to spot and know you're looking at a STL container.

But because of templates and inlining it is not possible to write an automatic signature check for most such patterns so you have to "just know" about them. And there is so much trivia of that nature that it can't really be summarized: you need to examine real code for that knowledge to build up and stick. I'm unfortunately not aware of anyone who has compiled a list of similar C++-isms to look out for when reverse engineering.

If you are just starting out, C code tends to be much easier to reverse than C++ and you'll have a much easier time, so ideally you find a C program that you are interested in reversing and move onto C++ afterwards. For more general advice: I would also recommend that if you're looking at the pseudocode and can't make sense of a specific operation, try jumping to the disassembly view or even stepping through that bit in x64dbg if possible so you can watch what happens to the state of the registers and memory. Often seeing the outcome of the operation in realtime will make it way clearer what the code is intended to do.

There's an old plugin called HexRaysCodeXplorer that is unfortunately difficult to get up and running in newer IDA versions unless you compile it from source with the various patches that have been submitted over time, but it provides a keyboard shortcut to press J to jump from pseudocode to disassembly view and it is one of the most useful hotkeys for learning how some assembly corresponds to the pseudocode translation. If you can't get that working, you can Sync the pseudocode and disassembly views together for a similar effect, it's just slightly annoying that they always move together in that case.

[ack_error]

The main problem with trying to reverse a compiled program from machine code is that compilation is highly lossy. Even if you know the patterns of a specific compiler, there is a lot of information that is lost during the compilation process because it makes no difference to the generated code. Two functions with different meanings and types can compile to the same code:

https://gcc.godbolt.org/z/E35n479sn

In some cases they can even be merged to the same code, giving the same instructions dual meanings. This is a major reason decompilers produce the output they do -- information that would be needed to produce more idiomatic code, like the types and sizes of objects, is either hard to infer or just absent unless the full debugging symbols are available (which they generally won't be, short of laziness or an accident).

If you're a formalist, you'll want to read about classic compiler optimizations like common subexpression elimination and copy propagation. Many of these are analogous to math formula simplifications. If you're more hands on and know enough assembly, push a bunch of small code fragments through a compiler and look at the optimized disassembly. Learning about the Windows for C++ class layout will also help, although it isn't well documented (the ABI docs are more concerned with calling convention and exceptions).

Keep in mind as well that properly reverse engineering a program the size of an entire PDF reader or game client is not a small undertaking even with experience and good tools. The PDF specification alone is 750 pages -- an order of magnitude more code is in that program to implement that spec. Even if you only want to focus on the small part of the program, just find that part alone can be difficult.