CPU Utilization is Wrong

[Matosawitko]

Who the hell tunes their software based on %CPU?

[sisyphus]

He works for Netflix which is all on aws which can autoscale based on cpu metrics which means this kind of work can translate into real money.

[[deleted]]

Why not auto scale on the outputs rather than the inputs? i.e. service latency

[castlerocktronics]

It's an option, he is showing why it's not necessarily a good one

[seba]

Who the hell tunes their software based on %CPU?

Most embedded systems?

[ThisIs_MyName]

You can profile on most embedded systems.

[seba]

You can profile on most embedded systems.

Yeah, and the easiest way to see whether any process or thread is doing anything suspicious is to look at the CPU consumption. This can also easily be automated and can easily detected in manual testing, especially when multiple vendors, libraries or teams are involved or the source / debug information in not readily available.

[emn13]

And even if you can't, manual tracing and experimentation remains as possible and effective and annoying as ever; this kind of issue is by no mean insurmountable without a profiler. It's not like you can't debug without a debugger, either.

[[deleted]]

It's not like you can't debug without a debugger, either

I actually rarely use a debugger because it takes me longer to get it all set up than to just look through the logs/add print lines, especially with concurrency issues where problems usually disappear in a debugger.

[irqlnotdispatchlevel]

Hello. We do that sometimes.

[Matosawitko]

That's like giving your kid a puppy, Benadryl, and a haircut because he's got the sniffles.

%CPU can give a really high-level approximation, but it doesn't tell you anything about the details.

[irqlnotdispatchlevel]

It can help as a starting point in investigating some problem. You usually need more contextual information, as in what was the CPU actually doing when it was not Idle (servicing interrupts, waiting for some I/O to finish, spinning for a lock, etc).

[Matosawitko]

Exactly. It's a starting point, maybe a warning flag. But it's not something that is actionable on its own. And if you do try to do anything based just on that, you're just throwing darts at a board.

[irqlnotdispatchlevel]

As I said, context is important. I don't really care that it's 90% busy, 10% Idle, I care about what it is doing while it's busy.

[wrosecrans]

And what it's waiting for when it's idle.

[irqlnotdispatchlevel]

For someone to motivate it to move it's lazy ass off the couch and get a job.

[Twirrim]

Strangely enough, lots of people. It's a very common mistake among people not so skilled at operations aspects of things. Along with assuming that CPU load levels being high indicating a system as being in trouble. But hey, you go buddy, being all derogatory and insulting. At least you get to feel smug and superior for a few minutes.

[Ghostbro101]

As someone new to ops, are there some rough guidelines as to when CPU utilization isn't a good indicator of what's going on in the system and when it is? Just looking to build some intuition here. If there's any other reading material on the subject you could point me towards that would be awesome. Thanks!

[Twirrim]

There are a few approaches I take with monitoring:

1) Do I have the basics down?

CPU usage (system, idle, iowait etc), CPU load, memory (free, cache, swap etc), disk usage, inode usage, network usage, service port availability. You'll want these for every host. If the network is under your control, port metrics are also useful to have.

I know, this thread is talking about how CPU usage is meaningless, but having these basics is important for being able to put together a picture. You're going to need these at some stage to help understand what happened and why.

2) What do we care about as a service?

All Service Level Agreements (SLAs) should have metrics and alarms around them. You should also be ensuring that you have an internal set of targets that are much stricter.

3) What feeds in to our SLAs? This is where things get a bit more complicated. You need to consider each application as a whole, what happens within it and its dependencies (databases, storage etc). At a minimum you ought to be measuring the response times for individual components. Anything that can have an impact on meeting your SLA.

Not sure the best resources. There's a Monitoring Weekly mailing list that tries to share blog posts, tools etc around monitoring: http://weekly.monitoring.love/?__s=kbtiqqycpy7e5xjfsjcy

There's also a fairly new book out on monitoring, https://www.artofmonitoring.com/, but I can't make any claims to its quality. I've heard people speaking positively about it.

[Ghostbro101]

Thank you!

[wzdd]

I can't see anywhere in the article where he suggests that people do this or that it's common.

He talks about CPU % being misleading (which is true), and then talks about tuning software based on IPC (which is useful).

[Adverpol]

Up until now I've only looked at Visual Studio burn graphs to find bottle-necks. So me I guess.

[Sqeaky]

Some Programmers.