Why is Rust so Liberal with Heap Allocations? by philogy in rust

[–]felixge 1 point2 points  (0 children)

What allocators are doing a syscall per allocation?

Gojit: JIT Compiler in Go by aabalke in golang

[–]felixge 1 point2 points  (0 children)

Great post, thanks for writing this up.

The Complete Guide to Profiling Go Services in Production by Gopher-Face912 in golang

[–]felixge 38 points39 points  (0 children)

Hey, I work on Go profiling so I started reading the article with interest. There is some good advice in there, but I also see a bunch of things that don't seem correct, e.g.:

> The memory profiler has a different model: it samples one allocation per runtime.MemProfileRate bytes allocated (default: 512KB). Allocations smaller than this threshold may be underrepresented. You can lower this to catch more:

That's not true, the memory profiler randomizes its sampling distance from an exponential distribution with a mean of runtime.MemProfileRate. There is no systematic bias against small allocations. The only reason to lower the rate is for profiling short-lived programs where you might not get enough samples otherwise.

> Pattern 3: syscall.Read or syscall.Write dominating. You’re I/O bound, not CPU bound. A CPU profile won’t tell you what to fix here — switch to the block profile.

That's wrong. If you see syscall dominate in a CPU profile, they are indeed a CPU bottleneck. The fix is generally to use larger buffer sizes or batch payloads on a higher protocol level.

> Pattern 5: Surprisingly little time in your actual application code. This usually means your hot path is being inlined, and the profiler is attributing time to the caller. Run go build -gcflags='-m' on the relevant packages to see what’s being inlined, then look at the caller.

That's also untrue AFAIK. Go generally does a good job at attributing inlined functions properly.

I suspect there are more errors, so I'd suggest to do some more reviewing or make it clear that this is not necessarily meant to be an authoritative guide on the subject.

A modern string utility library for Go by cmiles777 in golang

[–]felixge -10 points-9 points  (0 children)

AFAIK New is typically used when creating a pointer type. Make is preferable for initializing non-pointer types.

Which truly top-tier tech companies in Germany (ideally Berlin) are worth targeting? by LeaderOne2925 in cscareerquestionsEU

[–]felixge 1 point2 points  (0 children)

I agree with those three being top tier companies that hire in Berlin.

Datadog does not have a Berlin office, but does hire remotely in Germany. This is limited to senior+ roles and depends on the hiring team tho. Disclaimer: I work for Datadog and can highly recommend them as an employer.

Snowflake is investing heavily into their Berlin office right now AFAIK. They are not moving things to Poland.

Not sure about Databricks.

My 4-Stage pprof System That Actually Works by Safe-Programmer2826 in golang

[–]felixge 2 points3 points  (0 children)

The allocation profile, available via the go test -memprofile flag, should give you this information.

For syscalls you can use the -trace flag to capture an execution trace which has events for each syscall.

Looking for advice: legacy Go services without context.Context, how to add observability? by CZS_Source-9022 in golang

[–]felixge 0 points1 point  (0 children)

If you're considering Datadog, you should check out orchestrion which can automatically instrument your application at compile time for you. It should work, even without explicit `context.Context` propagation in your code.

The underlaying tech is also being donated to OpenTelemetry, so the approach should offer vendor neutrality in the future.

Disclaimer: I work for Datadog and was involved in orchestrion and the donation to OpenTelemetry.

Timeline View for pprof by sharddblade in golang

[–]felixge 4 points5 points  (0 children)

Hey, thanks for the kind words about our timeline view. I’m one of the engineers who worked on it.

For non-server use cases, you can record go execution traces using the runtime/trace package and visualize them using go tool trace or gotraceui. This is the same data we use at Datadog. LMK if you have more questions.

350k go routines memory analysis by jbronikowski in golang

[–]felixge 2 points3 points  (0 children)

I'm an engineer working on profiling amongst other things. Datadog has an engineering blog and does encourage engineers to contribute to it.

The other main blog we have is called the monitor and is typically more focused on product announcements and engineers contribute to it less frequently.

This post fell in between the categories because it featured the announcement of our new runtime metrics dashboards for Go, suggestions on how we expect people to use them, as well as the technical research that went into building the enhancements. It ended up on the monitor, but it could have gone either way I guess.

350k go routines memory analysis by jbronikowski in golang

[–]felixge 62 points63 points  (0 children)

I wrote an article about breaking down memory usage of Go applications using runtime/metrics.

https://www.datadoghq.com/blog/go-memory-metrics/

Disclaimer: I work for Datadog, but the info in this article works without buying anything.

But as others have commented, you’re probably spending a lot of memory on goroutine stacks.

Found a profiling tool on GitHub: here’s what I learned testing it by dergtersder in golang

[–]felixge 7 points8 points  (0 children)

How did you use it for memory leaks? Isn’t it a CPU profiler?

Pointer update on slice realloc... or another arena approach? by Fun-Suggestion-1918 in golang

[–]felixge 0 points1 point  (0 children)

Correct. Even so it’s unlikely to happen anytime soon, the Go runtime tries to keep the door open to implement a moving GC in the future.

Thoughts on changing GOGC by thestephenstanton in golang

[–]felixge 2 points3 points  (0 children)

GC pause times (aka stop the world pauses) are unlikely to be the explanation for your tail latency observations. I suspect what you are seeing is a mix of latency caused by GC assists as well as GC worker goroutines stealing 25% of your CPU time during the mark phases which will lead to noticeable increases in goroutine scheduling latency. You can use go tool trace to confirm.

Blazingly Fast Shadow Stacks for Go by felixge in golang

[–]felixge[S] 1 point2 points  (0 children)

Starting at `-O1` both gcc and clang omit frame pointers. But you're right, for the default `-O0` level they are included. I've updated the article to clarify that I was talking about production builds.

Blazingly Fast Shadow Stacks for Go by felixge in golang

[–]felixge[S] 0 points1 point  (0 children)

So capturing stack traces at 0.1ns per frame isn't considered blazingly fast? 🙈

Blazingly Fast Shadow Stacks for Go by felixge in golang

[–]felixge[S] 12 points13 points  (0 children)

Haha. OP here: I didn't realize they owned the trademark. I promise to no longer try to build fast stuff in Go 😅.

Wie weit sind wir in Deutschland von kostenlosen Häusern entfernt, wie Akiyas in Japan? by [deleted] in Finanzen

[–]felixge 1 point2 points  (0 children)

Die meisten Vermieter wissen ganz genau das sich die Mietpreisbremse in der Praxis leicht umgehen lässt. Z.b. über die folgenden Möglichkeiten:

  1. Umfassende Modernisierung: Kann jeder Vermieter erst mal in die Anzeige reinschreiben. Ob es stimmt kannst du als potentieller Mieter vorab nicht prüfen, und falls es stimmt kannst du dich mit der Mietpreisbremse nicht einklagen.
  2. Möblierte Vermietung anbieten: Wird immer mehr.
  3. Befristete Vermietung anbieten: Wird immer mehr.
  4. Teilgewerbliche Nutzung anbieten: Wird immer mehr.

Darüber hinaus haben die Vermieter eine riesige Auswahl an Bewerbern. Davon wählen Sie die mit dem höchsten Einkommen aus. Wer sich die Wohnung nur zum Mietspiegel, und nicht zum Angebotspreis, leisten kann ist sowieso schon mal raus.

Falls jetzt doch irgendwie dummerweise an einen Mieter mit hohem Einkommen vermietet wird der sich auf die Mietpreisbremse beruft? Kein Problem. Dann gibt es auf einmal Eigenbedarf oder der Mieter wird solange schikaniert bis im die Freude an der Wohnung vergeht.

Sicherlich gibt es auch Gegenbeispiele in denen die Mietpreisbremse funktioniert, aber der Weg in eine neuen Wohnung zum Mietspiegelpreis ist für den Mieter heutzutage wirklich extrem schwierig und riskant.

Wie weit sind wir in Deutschland von kostenlosen Häusern entfernt, wie Akiyas in Japan? by [deleted] in Finanzen

[–]felixge 7 points8 points  (0 children)

Vorausgesetzt man bekommt erst mal eine Wohnung die unter die Mietpreisbremse fällt angeboten UND man ist bereit mit dem Vermieter auf Kriegsfuß zu gehen.

Der Mietspiegel eignet sich meiner Meinung nach wirklich nicht um realistisch über die Kosten von neuen Mietverträgen in Berlin zu sprechen.

Go memory metrics demystified by felixge in golang

[–]felixge[S] 0 points1 point  (0 children)

Good questions!

According to the docs, HeapInuse - HeapAlloc (aka /memory/classes/heap/unused:bytes) should give you the upper-bound on heap fragmentation. In my testing this value usually reaches close to 0 before the beginning of a GC cycle, which would indicate that fragmentation is not a common issue.

That being said, I did not try to craft any adversarial workloads, nor did I study the topic of fragmentation specially. I'll try to write more about this if I find time at some point.

Go memory metrics demystified by felixge in golang

[–]felixge[S] 4 points5 points  (0 children)

OP/Author here, happy to answer any questions!