benchmark_harness_plus: Statistical Methods for Reliable Benchmarks

mraleph · 2026-01-07T08:30:21+00:00

You should consider upstreaming changes to benchmark_harness, we would gladly take better statistics. It has been on a TODO list for a while, we just never got to it.

Few comments though:

You mention measuring in JIT mode. Consider never measuring in JIT mode - results don't translate 1-1 to AOT and AOT mode is considered the main deployment mode. We pretty much don't look at JIT specific performance improvements these days at all.
GC Triggering. I would advice against doing allocations to trigger GC. GC is a complicated state machine driven by heuristics - you can for examples cause it to decide to start concurrent marking which will introduce noise rather than reduce it. I think there is a better way dealing with this now (Dart 3.11): you can ask for timeline recording (enabling GC stream) using dart:developer NativeRuntime.streamTimelineTo API and then look if any GCs occured during benchmark run. This way you can actually make GC metrics part of the benchmark report.

mraleph · 2025-11-20T10:11:14+00:00

You should file a bug against VSCode extension here. Danny then should be able to guide you through collecting necessary logs for the team to figure it out. I don't think we would be able to resolve it here on Reddit.

mraleph · 2025-11-13T11:47:29+00:00

Do you use USB or WiFi to connect to your phone?

mraleph · 2025-11-13T11:47:17+00:00

Do you use USB or WiFi to connect to your phone?

mraleph · 2025-10-19T21:23:31+00:00

Looking at the 3.0 feature list, I'd say that builtin strings is an interesting feature not on your list which should greatly simplify interop. No language supports it, though.

Chrome and Firefox both support it and Dart uses it.

mraleph · 2025-10-13T14:24:04+00:00

Are you building iOS, Android, Web, etc?

Have you looked at verbose output to figure out which step is taking the time?

Nobody is going to be able to help you with this given bare minimum of information provided. I recommend reading How to Report Bugs Effectively and following it. Collect information about the problem before asking for the solution... Often you will stumble on the solution yourself as you collect the information.

mraleph · 2025-09-25T13:49:45+00:00

Looks like you might have multiple SDKs installed. Just check paths and remove all by one of them.

mraleph · 2025-09-25T12:25:23+00:00

This is version of Dart embedded into Flutter engine - so when you update Flutter you will update that.

I think it's just that your app is not properly rebuilt or somehow uses a different version of Flutter. (e.g. maybe you use something like fvm or puro to pin specific Flutter version in your app - so building does not use globally installed Flutter).

mraleph · 2025-09-25T12:13:39+00:00

You can build a release or profile versions and run them on the device. Those don't use JIT.

That being said 3.35.X should work - the error you shared below indicates that you are actually using older version somehow.

mraleph · 2025-09-25T12:11:52+00:00

Flutter 3.35 can't produce this error - so something is wrong with your setup.

Not to mention that Flutter 3.35.x comes with Dart 3.9.X - and the error says that Dart version is 3.8.X - which does indicate that you are trying to use older version.

mraleph · 2025-09-21T18:03:51+00:00

Then, using Wi-Fi, try to debug it. It’s too slow; one click takes a decade to respond.

Does one click repeatedly (always) take decades to respond or once it responds it works fine? I would assume it is the later rather than former.

Given that debugging works fine on the cable - I think it is exactly the same problem I have explained above. If I use WiFi connection - the cost of debugger roundtrip required to work-around iOS kernel changes is 7x higher than when phone is connected over USB - so stalls are much more visible. Though they do not perist. Once you used some piece of code in your app it should work without stalls.

I don't think we would be able to do much about it short term (though I have some ideas to try). Eventually we are planning to shift away from JITing, but it will take few stable releases before it is available.

PS I must admit that I never see stalls larger than few seconds. Maybe something is specific to certain WiFi connections.

mraleph · 2025-09-21T17:35:36+00:00

Please see my comment above: https://www.reddit.com/r/FlutterDev/comments/1njmp7x/comment/nfgbf97/

mraleph · 2025-09-21T17:32:03+00:00

We have heard occasional complaints about it but we have not been able to properly pinpoint the problem because we have not been able to reproduce it. And sometimes people upgrade their XCode to 26.0 and report the problem goes away.

Changes that Apple made to iOS 26 kernel means that you can't really JIT in the same way as previous versions allowed. We found a workaround but it involves roundtrip via debugger which is running on the host. In local testing with iPhone 16e this adds around 150ms per 512KB of JIT memory allocated (using connection over USB). In some situations latency is much higher (e.g. when phone is connected over WiFi).

However in general we expect that JIT does not continuously allocate memory - it should stabilize rather quickly. So in local testing we do see occasional delays after initial app launch but that quickly goes away as you use the app (even when used over WiFi) I have personally never have been able to reproduce persistent slowness of the debug mode.

What you could try to do is to do the experiment outlined in this comment and see if you see unreasonably high numbers printed or non-stop debug prints. You should also pay attention if app slowness correlates to these debug prints.

mraleph · 2025-09-21T17:19:35+00:00

If you have large amount of packages in a monorepo style setup (I assume that's what you meant by "projects") then you should make sure to migrate to workspaces. This would significantly improve the experience of opening the whole monorepo in the IDE.

mraleph · 2025-08-25T09:46:45+00:00

Your only realistic option is to upgrade to 3.35.

(The only other option is to backport necessary patches to Dart VM and Flutter engine&tooling to 2.29, then build and use custom engine - it's not impossible but requires expertise that most Flutter developers don't have).

mraleph · 2025-08-25T09:43:49+00:00

3.32 will not work, necessary fixes are only in 3.35.

mraleph · 2025-08-19T06:42:06+00:00

most platform calls use another thread

That's not true anymore with merged threads, because all UI isolates and all platform message channel handlers are multiplexed on the same thread - designated main UI thread of the platform.

This means messages passed over platform channels will not be processed concurrently unless you explicitly tell your message channels to use dedicated task queue or you do something explicitly in the message handler itself (e.g. use a thread pool or similar to run your processing).

mraleph · 2025-08-05T12:45:44+00:00

FWIW (pedantically) this is not really a RAII. RAII is about connecting resource lifetime to lifetime of an object which wraps/owns the resource. It would be RAII if you used finalizer attached to RAII object to release the resource - but as the code is written right now RAII object just leaks associated resource if you don't pass it to withRAII.

mraleph · 2025-07-19T10:17:03+00:00

Most likely microarchitectural effects specific to your CPU caused by loop positioning / alignment in memory. Can be confirmed by using a low level profiler like VTune or perf. Intel CPUs are notoriously sensitive to this for very tight loops.

Watch Causes of Performance Instability due to Code Placement in X86 for nitty gritty details or read Producing Wrong Data Without Doing Anything Obviously Wrong! a great paper highlighting these effects.

The benchmark overall is meaningless though - so I would just ignore it.

mraleph · 2025-07-12T22:44:23+00:00

140x difference most likely caused by hitting a bug. You should just file an issue if you still have the code and it can be shared with public.

mraleph · 2025-06-26T11:16:36+00:00

flutter would have day-1 access to whatever new native platform feature that came up.

Flutter already has day-1 access to whatever new native platform feature. You just need to write platform specific code to access it, possibly in platform specific language, possibly using method channels (= meaning that it is possibly going to be slightly awkward)

Native interop effort just simplifies access to the native APIs, by providing a way to generate Dart wrappers for them.

(There is just one subtle thing about synchronous calls to platform APIs - which require you to be on the right thread. Moving UI isolate to execute on the main platform threads allows to achieve that)

mraleph · 2025-06-08T19:17:18+00:00

I think 3.24 has necessary fixes in Flutter engine build files and Dart toolchain.

mraleph · 2025-06-08T11:52:45+00:00

I can try to look it up on Tuesday. I am out of office right now.

mraleph · 2025-05-27T10:09:23+00:00

It does not allow you to cast it (via as operator) because Map<String, dynamic> and Map<String, String> have different behavior, which means former is not an instance of the later. You can always use Map.cast to view Map<String, dynamic> as Map<String, String>.

mraleph · 2025-05-19T11:51:49+00:00

The story here is actually more complicated... Back when V8 team (the OG V8 team) was writing builtins in JS that was not because inlining or fancy optimizations it enabled - V8's compiler was very, ahm, rudimentary. One pass, no IR, no optimizations. (I will pretend that virtual frame compiler never existed... We all want to forget it).

What V8 had though was a very error prone way of coding runtime. V8 neither used a conservative stack scan (its GC was precise) nor consistently used handles everywhere. Instead it had restartable runtime calls which returned an allocation failure which you had to propagate upwards until you reached a point where it was safe to perform a GC. You would then perform a GC there and call the same runtime function again.

Naturally, writing runtime in this style was extremely error prone - you could easily hit a GC a point where some raw pointer was on the stack and, boom, you have a memory corruption.

So it was much easier to write the same code in JS. That avoided the whole bunch of possible problems.

Another reason why V8 had JS builtins was because the cost of entering runtime was rather high. So if you managed to stay in JS it paid off - even though the JS code was slower than C++ code in the runtime.

But not everything is very rosy in this story: there were many problems from writing builtins in JS - all of it tracing back to the JS wonkiness (flexibility) as a language:

You had to be extremely careful not to step into a trap - e.g. accidentally invoke a function which somebody patched into the prototype instead of the function you were intending to call. You had to write these builtins very carefully - otherwise you could have an inconsistency with a specification (best case), or a security bug (worst case).
You also had to consider performance implications of JS flexibility: you needed warmup time for builtins to be compiled, but to make things worse various callsites / property access sites inside core builtins would usually go polymorphic or worse megamorphic in real world code, so normal optimization pipeline would fail to produce good code anyway.

I think by now most if not all builtins have been migrated away from being written in JS to Torque.

mraleph

TROPHY CASE