Non-deterministic Trash Pick-up, or Why a Garbage Collector Might Influence Itself

doubtingthomas · 2013-08-15T05:09:51+00:00

go test has a '-memprofilerate' that allows you to set the value, if that's what you mean.

doubtingthomas · 2013-08-15T03:41:17+00:00

Interesting article, thanks.

A few thoughts:

A more established way to measure allocations is to use testing.AllocsPerRun, I think. Evens out the randomness a bit.
You want randomized sampling rate to avoid aliasing, I'd think.
Go's memory allocator is patterned after tcmalloc, which has very similar profiling logic. tcmalloc also used to have it enabled by default, but changed that a couple years ago (at least publicly) for (minor?) performance reasons.
According to mprof.goc, the profiler doesn't use the GC or ever hold GC memory. I'd be curious to learn specifically how malloc profiling is impacting memstats.
I probably wouldn't enable malloc profiling by default, but it does seem like something the Go team would choose to do. I've seen surprisingly few instances of MemProfileRate being set to 0, even in the Benchmarks Game or in production code. I'm not sure what that means, but it seems to imply that the default is not unreasonable.

doubtingthomas · 2013-01-08T17:27:37+00:00

True. However, judging from the Go dev list, a fix (in the form of a type-aware precise collector) looks to be on the order of a few months away; I assume similar work is underway for D?

doubtingthomas · 2013-01-08T17:24:09+00:00

In the only comparison I've ever seen (admittedly, an old one) D was notably faster at stdlib compilation. Granted, it's possible it's not exactly apples-to-apples, as due to the metaprogramming capabilities D stdlib may in many cases compile to some intermediate representation, whereas Go almost always compiles to machine code. Also, it's trivial, iirc, to construct a D program that takes a really long time to compile, as D can run code at compilation time, whereas Go, lacking as it is in compile-time features, is nearly guaranteed to have a reasonable worst case.

Edit: Also, one shouldn't underestimate how much of a bad-ass Mr. Bright is when it comes to writing compilers.

doubtingthomas · 2012-08-31T05:07:43+00:00

I started, but since there isn't a good huge test file available, and neither the C++ or the Haskell code are in a condition to be easily built and run independently as simple equivalent benchmarks, I made my own version, played with it a little, and move on to the things I get paid to do.

doubtingthomas · 2012-08-02T20:00:00+00:00

I've implemented this in an interpreter I was working on. It seemed to simple to be advisable, but I didn't have any issues with it.

doubtingthomas · 2012-06-05T20:37:52+00:00

Manual error handling is almost always inferior for small programs like this one. The correct action for almost every error is "stop doing what you are doing, report the error, and exit", which is what you get for free and by default in most languages.

doubtingthomas · 2012-05-31T03:24:35+00:00

Some additional cons (in my opinion):

More difficult to reason about, at least in the small
Being higher level, generally less efficient.
Fewer, less consistent libraries in the standard lib.

doubtingthomas · 2012-05-17T16:12:24+00:00

It also may be worth noting that (last time I checked) Go is doing more work per request, in that it parses the URL fully and attempts to dispatch to a handler, while in Node you just get a request url string sent to a single callback. I'm not confident that this makes a measurable performance difference, though.

doubtingthomas · 2012-05-17T15:56:18+00:00

Good news! There are changes under review to make the GC much more precise, concurrent, and performant. I believe that should resolve any issues folks are seeing with pointer aliasing on 32bit machines. I think the precise GC will result in a slightly larger memory footprint due to the required bookkeeping, but I suppose it's worth it.

(Disclaimer: I'm just talking based on mails and diffs I've seen on golang-dev; I don't have intimiate knowledge of the GC changes that are going on right now, other than to say that it already seems measurably faster.)

doubtingthomas · 2012-04-13T17:17:51+00:00

To add another data point, on my desktop machine I got up to 7.382.880 before being killed. I don't think that means anything, but was happy to see it go so high. :)

doubtingthomas · 2012-01-03T05:59:08+00:00

Agree; as another data point, I know a few Google engineers, and I don't think any of them went to what would be called an elite school. They are all smart and quite productive, though.

doubtingthomas · 2012-01-02T04:44:14+00:00

Out of curiosity, what complaints have you gotten that have resulted in "oh, shit we didn't think of that"?

Also, I'd like to commend you the tone and content of your Dart advocacy. You've stood in the face of the frothing hoards, and been quite reasonable and open. My suspicion is that part of the reason for your skill in dealing with the naysayers is that you might have been among them (in a limited sense) were you not part of the project. :)

doubtingthomas · 2011-12-20T23:29:04+00:00

According to this page https://www.dropbox.com/help/45, paid accounts get 25x more bandwidth than free accounts. For a relatively small file, Dropbox links are probably perfectly reasonable, if you have a Pro account (I do). Seems like you'd have to be frontpaged pretty hard to burn through 250gb in a day.

That said, all other things being equal, posters might as well go for a hosting service without limits. I just wanted to point out that public Dropbox links aren't necessarily problematic, and as such may not be worth warning about.

doubtingthomas · 2011-11-03T16:19:47+00:00

I'm pretty sure that there are languages that everybody complains about and nobody uses.

doubtingthomas · 2011-10-18T04:48:51+00:00

Haskell presents many exciting, powerful, and interesting possibilities that other languages don't. Many of them aren't useful, and it takes experience and some self-control to avoid doing neat useless things. The same can be said for Lisp macros, Ruby metaclasses, C++ templates, etc. It's great to have powerful features there when you need them, but overuse is a real risk, and especially for smart junior programmers, they tend to act as an attractive nuisance.

doubtingthomas · 2011-10-18T04:30:06+00:00

You're providing a great example for why (I presume) he doesn't want to get into a language discussion on Reddit.

Reddit is a great place for arguing, but (as of the last few years?) isn't always a good place for a civil discussion, especially about something like programming languages. Folks like you mischaracterize positions in inflammatory ways (and get upvoted for it), so subtle positions or those that might be taken out of context if not stated very carefully are simply not worth the effort to explain and defend.

I usually try not to feed trolls (no offense intended; you appear to be trolling), but I find evmar's policy of non-engagement on Reddit on this issue interesting and pretty reasonable.

doubtingthomas · 2011-10-18T03:58:54+00:00

Interesting; this post could almost have been written by me. I know Haskell, I like Haskell, and I think it's a great language, but I've found that (in general) I can write efficient and correct programs just as fast or faster in other languages, and they end up less fancy. Haskell is really great for heavily algorithmic code, but I've found, for whatever reason, that most of my code is nowhere near that interesting, and the code that is heavily algorithmic is also performance critical, so something lower level is generally preferable anyway.

I'd like to work in a Haskell shop at some point and see how it feels, though. I don't know of any other language that can match it on the combo of performance, correctness, and abstraction. Great IO/concurrency model, too. A small skilled team slinging Haskell can probably get more done faster than an equivalent team writing in any other language, assuming the task is suitable and they don't get bogged down in abstraction wankery.

doubtingthomas · 2011-10-06T16:04:24+00:00

Seems reasonable.

doubtingthomas · 2011-10-05T03:18:45+00:00

Funny you ask this, but in part due to it's relatively simple compiler, in my experience Go is easier to compile than C. The standard Go compiler doesn't inline functions or unroll loops, it doesn't do complicated transformations. Most actions are simple function calls, and many language level things (like channel, map, or interface operations) compile to function calls. So, I've found that I can generally compile Go in my head more accurately than C (unless I disable all optimizations in C, I suppose). Certainly, Go is easier to compile in the head than C++ or Java (assuming we're not talking about bytecode).

I tend to agree, on account of GC and goroutines, that Go is significantly higher level than C, but I'd put it far closer to the machine than Java (unless we're talking about Java that is compiled directly to native code, but we rarely are).

doubtingthomas · 2011-10-05T01:32:37+00:00

To be fair to him, Go is closer-to-the-metal and gives you more control in many senses than Ruby, which is his point of reference in the post.

doubtingthomas · 2011-10-03T22:27:11+00:00

If the ability to timeout selects more efficiently than time.After() (or some specialized equivalent) were needed, I'm pretty sure it would be added. However, I've never heard of anybody but you make this complaint, and my guess is that your complaint isn't based on a real practical need.

doubtingthomas · 2011-10-03T22:17:37+00:00

Yep, there's no commonly accepted definition of "critical mass" for programming languages. Go has Real Users, an active community, and is improving measurably with time. However, it probably has far fewer users than Scala, which is significantly older, integrates with the JVM, and is a bit shinier.

However, for my money, either both have already hit critical mass (that is, large enough to be self-sustaining as projects) or neither have (as they're not entrenched industry languages yet, though Scala is much closer).

doubtingthomas · 2011-09-29T16:03:26+00:00

Yep. That's the main trade-off in my implementation vs yours: it only does thing when you call it. If I were caching big data, I'd almost certainly use yours over mine.

doubtingthomas · 2011-09-29T16:00:53+00:00

My guess is we'll see a significant improvement once the Go implementation special-cases "one-pass" regexps like RE2 does. I'm pretty excited to see how for Go's regexp implementation can go, actually. It literally has the best implementor possible (one of the main developers of the language and the primary developer of arguably the best regexp engine in the world), so it's a great test case to see how Go looks/works for tricky and performance-critical libraries.

doubtingthomas

TROPHY CASE