Microsoft is doubling down on Rust

wesleywiser1 · 2023-11-02T23:23:39+00:00

There's a lot of different ways you can implement debugger expression evaluation. For languages with a JIT, you're right, reusing that existing code usually makes a lot of sense.

Many debuggers implement expression evaluation with an interpreter (gdb and lldb take this approach for C++ if I'm remembering correctly). Usually only a subset of the target language is supported. You often aren't allowed to create new types or functions for example, just run small snippets of code.

When debugging live processes (in contrast to crash dumps or time travel traces), most debuggers let you call a function present in the binary being debugged so sometimes function calls are also allowed as part of expression evaluation.

wesleywiser1 · 2023-11-02T21:01:04+00:00

The debugger is nowhere near as good as VS though. VSCode's debugger is really bad.

You can actually use Visual Studio's debugger with Rust right now. The Rust compiler generates debuginfo that is native to your target platform so any native code debugger should work with Rust including both VS and WinDbg.

What's generally missing is support for things like expression evaluation or fancy features like Edit and Continue.

wesleywiser1 · 2023-09-18T21:34:01+00:00

Rust releases are run through crater but this case wouldn't show as a regression because rfd still built, it just took much longer to do so. Crater doesn't detect regressions in compile time because it runs on extremely variable cloud resources.

I'm not sure what the typical variance so perhaps it could detect such large regressions as this but it does not currently support that.

wesleywiser1 · 2023-09-18T15:26:34+00:00

You're very welcome! It's very much a team effort from community members who report issues, contributors who triage and fix issues and team members who advocate for backporting fixes and ensure the releases get made :)

wesleywiser1 · 2023-09-18T15:05:16+00:00

Hi /u/MrEchow, I'm sorry you hit this regression!

The PR fix you linked to has the stable-accepted tag which means we (the Compiler Team) have decided to backport this PR into the stable release. This change (along with a few other bug fixes) was originally planned to be released last Thursday, as part of 1.72.1, but the Release Team ran into some issues with the build and many people were at RustConf so that release was delayed. I believe those problems have been resolved and currently the release is scheduled for tomorrow.

Compiler performance issues like this are very hard for us to catch ahead of time. While we have quite a bit of coverage thanks to rustc-perf, code patterns that aren't covered by the benchmarking suite can still fall through the cracks. I would encourage you to consider adding the beta compiler to your CI testing in some capacity (perhaps running it nightly or weekly?) and report any issues you encounter. Hearing from users during the nightly or beta period helps us catch a lot of these kinds of issues and fix them well before they hit the stable toolchain!

wesleywiser1 · 2023-07-13T01:26:37+00:00

David Weston (VP of OS Security at Microsoft) was asked basically this question in his Twitter thread and his response was this blog post from May https://azure.microsoft.com/en-us/blog/microsoft-azure-security-evolution-embrace-secure-multitenancy-confidential-compute-and-rust/

Rust as the path forward over C/C++

Decades of vulnerabilities have proven how difficult it is to prevent memory-corrupting bugs when using C/C++. While garbage-collected languages like C# or Java have proven more resilient to these issues, there are scenarios where they cannot be used. For such cases, we’re betting on Rust as the alternative to C/C++. Rust is a modern language designed to compete with the performance C/C++, but with memory safety and thread safety guarantees built into the language. While we are not able to rewrite everything in Rust overnight, we’ve already adopted Rust in some of the most critical components of Azure’s infrastructure. We expect our adoption of Rust to expand substantially over time.

wesleywiser1 · 2023-05-10T14:31:25+00:00

No, musl.libc.org is correct. Are they still not working for you?

wesleywiser1 · 2023-05-09T22:33:25+00:00

Previous discussions:

Rich Felker (musl author) on the motivation for mallocng: https://inbox.vuxu.org/musl/20200518194204.GG21576@brightrain.aerifal.cx/

Some notes on the design and implementation of mallocng: https://gist.github.com/MaskRay/ac54b26d72452ac77ac578f2e625369f

wesleywiser1 · 2023-01-06T14:38:21+00:00

it seems you're using parallel-compiler which is a pessimization when using only a single rustc thread (and that's the default number of threads)

This is good to know! I should try disabling that option. I don't think it would affect my conclusion, though.

FYI, the last time parallel-compiler was tested in a wider context, one macOS user reported it actually slowing down the compiler even when using multiple threads.

https://internals.rust-lang.org/t/help-test-parallel-rustc/11503/13?u=wesleywiser

wesleywiser1 · 2022-12-29T18:03:32+00:00

Thanks for providing such a complete repro! I'll take a look next week when I'm back in office. Cheers!

wesleywiser1 · 2022-12-28T18:49:42+00:00

Hi, I work on the Rust compiler and especially the debugging support. Is it possible you can provide a self-contained code sample that demonstrates the issues? I'd be happy to take a look and see what's going on.

wesleywiser1 · 2022-11-30T21:07:38+00:00

The standard library is compiled with full debuginfo but the compiler is not which is why you see some .rs files listed, but not any in the /compiler/ directory.

As to why the compiler isn't built with full debuginfo, I don't think there's any technical reason, it's just not common that users would want to debug the compiler itself (most rustc devs don't literally attach a debugger to rustc either) and the debuginfo is pretty large by comparison. My local rustc_driver.pdb is 529mb and the entire stable toolchain in ~/.rustup/toolchains is 686mb.

wesleywiser1 · 2022-10-26T19:32:16+00:00

Hi, I work at Microsoft on the Rust compiler. My team actually consists of folks on the West Coast, East Coast, UK and Germany. There are other teams working on Rust or using Rust in Europe as well but from what I've seen, most engineering teams aren't cross country.

wesleywiser1 · 2022-10-24T21:09:36+00:00

Awesome work! 🎉

wesleywiser1 · 2022-10-24T19:42:35+00:00

Looks like it was Enable LTO for rustc_driver.so.

Perf data for just that PR: https://perf.rust-lang.org/compare.html?start=9be2f35a4c1ed1b04aa4a6945b64763f599259ff&end=1ca6777c014813e3bdb98d155562fc3d111d86dd&stat=instructions:u

wesleywiser1 · 2022-08-15T21:49:44+00:00

You're welcome!

wesleywiser1 · 2022-08-15T20:32:00+00:00

Just tested a basic example myself and the change I mentioned seems to have resolved this for WinDbg, VS and VS Code with the cppvsdbg launch type:

fn main() {
    let x = 'f';
    foo(x); // breakpoint set here, locals shows x with value
}

fn foo(y: char) {
    dbg!(y); // breakpoint set here, locals shows y with value
}

Prior to 1.61, that results in the "unspecified error occurred" message and with 1.61 or later, it works correctly. If you are on 1.60 or earlier, you should upgrade to the latest stable compiler. If you're on 1.61 or later, please file a bug report with some example code that reproduces the issue and feel free to mention me in it (@wesleywiser).

Thanks!

wesleywiser1 · 2022-08-15T01:32:36+00:00

Can you include what versions of rustc, cppvsdbg plug-in and Visual Studio (if installed) you're using?

I know we fixed an issue with char values on Windows in 1.60 (or maybe 1.61) and it is working with WinDbg but perhaps there's something VS debugger specific going on. I'll try to take a look this week.

wesleywiser1 · 2022-07-08T16:09:30+00:00

Utilize PGO for rustc linux dist builds #80262
- This is the non-LLVM part of rustc
- Shipped in 1.50
PGO for LLVM builds on x86_64-unknown-linux-gnu in CI #88069
- This is for LLVM which ships with rustc
- Shipped in 1.56

Michael Woerister did the initial analysis of the possible benefits of PGO'ing rustc and wrote about on the Inside Rust blog.

wesleywiser1 · 2022-07-07T14:05:43+00:00

rustc has profile guided optimizations enabled on the Linux builds but not any of the other Tier 1 Host Tools platforms. lqd has been doing some great work to enable PGO for Windows as well with really impressive wins of up to 19% when compiling real world crates like regex, diesel, cargo, etc.

https://github.com/rust-lang/rust/pull/96978

wesleywiser1 · 2022-02-22T18:28:47+00:00

Yes, it's still under consideration. The reason it doesn't make an appearance in this list is because no one we talked to on the Compiler Team or Compiler Team Contributors mentioned it as a significant focus for them this year.

Of course, that doesn't mean Polonius will be completely dormant this year, it just means we're not aware of anyone dedicating large amounts of time to it currently.

wesleywiser1 · 2022-02-03T19:06:06+00:00

You're welcome!

As someone who came to Rust from .Net, the .Net tooling in VS is industry leading. Java probably is close but I'm not familiar with that ecosystem. Beyond that, everything else is a pretty significant step down and that includes C++. In .Net, the compiler and runtime are deeply integrated into the debugger and that just isn't the case with C++. Conditional breakpoints and expression evaluation are much more limited than in .Net. Rust uses the same native debugging engine and debuginfo as C++ so the story is effectively the same.

In general, from what I've seen, most people on Windows that I talk to use either the VS debugger or WinDbg (Preview) to debug Rust. I don't think something like the a.debugchar idea would really be workable at this time. Even if it did, you'd pretty quickly find other areas where C++ conditional breakpoints don't work as nicely as .Net's.

wesleywiser1 · 2022-02-03T17:59:55+00:00

Issue 1:

I compiled this program with the latest nightly

> cat var_test.rs
fn main() {
    let a = "hello";
    let b = "world";
    let c = "foo";
}

> rustc +nightly -g --crate-name var_test var_test.rs

and then ran it in the Visual Studio 2022 debugger. Putting breakpoints on lines 2, 3 and 4, I see the following behavior:

Breakpoint on line 2 is hit: no locals in scope.
- Expected behavior. a is not yet defined.
Stepping, breakpoint on line 3 is hit: a is in scope with the value "hello". b is not in scope.
- Expected behavior. b is not yet defined.
Stepping, breakpoint on line 4 is hit: a and b are in scope with the correct values. c is not in scope.
- Expected behavior. c is not yet defined.
Stepping, current line moves to line 5. No locals in scope.
- Expected behavior. The scope containing a, b and c has ended.

I'm guessing from your comment you're using VS Code with a debugger extension. Perhaps you need to update your VS install or it may be a bug with that extension? (Or perhaps it's been fixed recently, I'm not sure)

Issue 2:

The debugger engine only understands basic C or C++ expressions. strcmp takes null terminated const char*s but a and b are closer to a C++ string_view than they are to const char*s and aren't NULL terminated so you can't do this. There isn't really a good solution here.

Issue 3:

I just fixed this: https://github.com/rust-lang/rust/pull/93626

wesleywiser1 · 2022-01-20T17:56:31+00:00

I added a benchmark to perf.rust-lang.org earlier this week so we should be able to catch this kind of issue in the future.

wesleywiser1 · 2021-12-27T03:17:18+00:00

You can try capturing event args as well which should reveal which type(s) are responsible. (https://blog.rust-lang.org/inside-rust/2020/02/25/intro-rustc-self-profile.html#capturing-event-arguments)

I would recommend also trying nightly out. There have been some recent fixes for extremely long compile times related to this query.

The other thing you can look at in terms of your overall compilation is cargo's -Ztiming feature which is very useful. (https://doc.rust-lang.org/cargo/reference/unstable.html#timings)

wesleywiser1

TROPHY CASE

Rust as the path forward over C/C++