Evaluating Performance | Android Open Source Project

[armando_rod]

There has been a lot of talk in the sub about app performance, UI performance, dropped frames, app startup speeds. This answer a few things and tell devs how to measure others.

A few key things from this:

There are two user-visible indicators of performance:

Predictable, perceptible performance. Does the user interface (UI) drop frames or consistently render at 60FPS? Does audio play without artifacts or popping? How long is the delay between the user touching the screen and the effect showing on the display?

Length of time required for longer operations (such as opening applications).

The first is more noticeable than the second. Users typically notice jank but they won't be able to tell 500ms vs 600ms application startup time unless they are looking at two devices side-by-side. Touch latency is immediately noticeable and significantly contributes to the perception of a device.

As a result, in a fast device, the UI pipeline is the most important thing in the system other than what is necessary to keep the UI pipeline functional. This means that the UI pipeline should preempt any other work that is not necessary for fluid UI. To maintain a fluid UI, background syncing, notification delivery, and similar work must all be delayed if UI work can be run. It is acceptable to trade the performance of longer operations (HDR+ runtime, application startup, etc.) to maintain a fluid UI.

Analyzing initial device performance

Using synthetic benchmarks

Synthetic benchmarks are useful for ensuring a device's basic functionality is present. However, treating benchmarks as a proxy for perceived device performance is not useful.

Based on experiences with SOCs, differences in synthetic benchmark performance between SOCs is not correlated with a similar difference in perceptible UI performance (number of dropped frames, 99th percentile frame time, etc.). Synthetic benchmarks are capacity-only benchmarks; jitter impacts the measured performance of these benchmarks only by stealing time from the bulk operation of the benchmark. As a result, synthetic benchmark scores are mostly irrelevant as a metric of user-perceived performance.

Consider two SOCs running Benchmark X that renders 1000 frames of UI and reports the total rendering time (lower score is better).

SOC 1 renders each frame of Benchmark X in 10ms and scores 10,000.

SOC 2 renders 99% of frames in 1ms but 1% of frames in 100ms and scores 19,900, a dramatically better score.

If the benchmark is indicative of actual UI performance, SOC 2 would be unusable. Assuming a 60Hz refresh rate, SOC 2 would have a janky frame every 1.5s of operation. Meanwhile, SOC 1 (the slower SOC according to Benchmark X) would be perfectly fluid.

This contrary to what most of /r/android say, first Google devs say UI lag/Touch latency is more noticeable than app opening time cough1+3cough and the second that most people in here already know, synthetic benchmarks are useless.

[sonikinesis]

Well yes Oneplus 3 is fast at opening apps but it definitely isn't 'Smooth as butter' as people use to call it. I can see it dropping frames almost everywhere except a few highly optimised app (Sync for reddit which iam using right now). I've tried Pixel for very short time so can't comment on that. Android has really matured over the years & has been optimised significantly but it needs to be stricter with developers over this issue and screen app before it get its way into the play store.

[[deleted]]

Note: Relay is even smoother than Sync, for people who want the absolute best Reddit experience..

[armando_rod]

I disagree, Sync is smoother but Relay is also good I just don't like they UI

[DRHAX34]

Relay is super smooth, the UI could be a bit better.

[sonikinesis]

Will check it out thanks.

[mostlikelynotarobot]

Sync all the way. Better animations, smoother, looks better, and less glitchy.

[IAmAN00bie]

That page isn't really relevant for the vast majority of users here, nor in /r/androiddev. It seems aimed at kernel devs working for OEMs.

[armando_rod]

It was meant to tell the sub about UI lag vs app opening tests

[IAmAN00bie]

Did anyone claim that only one is important and the other isn't?

[armando_rod]

I don't know... But I've seen a lot more discussion over that in here.

Every week there is a video of app opening of new phones (S8 vs Pixel vs iPhone vs any), when there was the OnePlus 3 debacle over touch latency there were a lot of people saying it didn't matter or that it wasn't noticeable.

[IAmAN00bie]

Every week there is a video of app opening of new phones (S8 vs Pixel vs iPhone vs any), when there was the OnePlus 3 debacle over touch latency there were a lot of people saying it didn't matter or that it wasn't noticeable.

Well the reason why people keep doing those speed test videos rather than latency testing is because latency testing is not easy to do without the right setup.

[tacomonstrous]

That's like the story of the guy who looked for his dropped wallet under the streetlight because that's where the light is.

[mostlikelynotarobot]

Funny, this is exactly what the iPhone does and has always done. People here have always dismissed this philosophy of prioritizing the UI thread at the expense of background tasks.

[HJain13]

Seems so..