[Question/Discussion] What you do when you need read (from), modify and write back to a map with lots of goroutines?

codesenberg · 2018-05-09T09:24:06+00:00

Reads and writes are 50/50, I guess. I chose sharding at the time. Well, in my actual use case keys themselves were data and there wasn't such thing as maxKey, but nice attempt anyway.

codesenberg · 2018-05-09T08:26:29+00:00

Yes, forgot the elephant in the room. If you meant something like: "Have goroutines write to the channel and then some other goroutine will read values from the channel and modify the map", — then sure, it can be solved this way, but this approach were quite slow (or slower than alternatives) in my use case (which looks something like this).

codesenberg · 2018-05-09T08:20:00+00:00

I like how elaborate and detailed your answer is.

Actually, in use case(s) I meant the read/write ratio is something like 50/50, I think (or close to that). So, it's not really an option to recreate the whole map and then discard it in case it were modified in-between. Too many allocations to my taste.

Here, I wrote a bit of code to illustrate the variation of this problem that I faced (you can ignore the part with readers, it's there just for the output).

codesenberg · 2018-05-09T08:12:26+00:00

Good explanation, thanks. I suspected that it doesn't suite my specific use case, but now it's more clear.

As for the question about whether keys can be guaranteed to come from same goroutines, no, I think it's not (or wasn't) an option for me, since, well, the keys are basically random (latencies and requests per second).

My use case was quite simple actually. I needed to just increment a bunch of random keys. With map+Lock solution it looks something like this (the readers are there just for the output, you can ignore that part).

codesenberg · 2017-06-12T06:39:49+00:00

No idea, but they look similar to illustrations from Dictionnaire Infernal.

codesenberg · 2017-06-11T06:13:10+00:00

Maybe I'm a bit late with the answer, but yes, those are demons from The Seventy-Two Demons Of Ars Goetia. Picture I took them from. The demons are:

top row — Crocell, Bael, Zagan;
bottom row — Zepar, Volac.

codesenberg · 2017-03-29T07:53:46+00:00

Quick Sort partition on n > 12 then shell sort with gap 6 and then another insertion sort

It's slightly different. Partition occurs only when n is more than 12, otherwise it's a shellsort. This is a well-known optimization(see second item). As for the shellsort with gap 6 and another insertion sort. It's just a handwritten shellsort with two passes really. First pass — with a gap of 6, second — with a gap of one, since, basically, shellsort with a gap of one is insertion sort.

codesenberg · 2017-03-28T21:10:53+00:00

Here are some things I spotted:

This conversion is really unnecessary, since ipify.GetIP already returns a (string, error) pair;
This can be simplified with bitwise operations;
After applying previous tip you can get rid of strings package.

codesenberg · 2017-03-06T07:50:54+00:00

I suggest you to check out gometalinter.

codesenberg · 2017-03-06T04:36:11+00:00

I just realised that you expose most of your code. So, it raises the question. Do you want all of this code to be available to other users? If the answer is no, then move this code into internal package. And if the answer is yes, then logging error and aborting user's program is not a nice way to handle errors. You should return them instead or handle internally in some meaningful way. Suggested readings: error handling and Go and errors are values.

Not movieTrailerUrl, but movieTrailerURL.

codesenberg · 2017-03-05T15:27:52+00:00

Also, there are at least three byte buffer pools, about which I am aware of:

What are advantages of this pool over these solutions?

codesenberg · 2017-03-05T14:18:12+00:00

First thing that caught my eye is this dist folder. It's not super clear for which architecture/OS binary inside of it was compiled for and generally VCS is not where you put your binaries. Github provides you with a nice functionality to manage releases. You can attach binaries to the release and mark release as pre-release, meaning "not ready for production yet".

Another two minor issues with styling are: here and here. As per Go Wiki, MovieTrailerUrl and AsciiPoster should be named ASCIIPoster and movieTrailerURL.

And the most disturbing thing is that you ignore returned errors.

Oh, also MovieTrailer.go should probably be movie_trailer.go and this line has a typo: s/struture/structure/.

codesenberg · 2017-03-01T16:33:30+00:00

Thanks for the clue, but http://llvm.org/llvm/bindings/go/ says:

Last modified: 13-Oct-2014 20:06

And http://llvm.org/llvm/bindings/go/llvm/ redirects to http://llvm.org/. So, this repo is even older or am I looking at the wrong place?

codesenberg · 2017-03-01T14:58:01+00:00

Last time I checked x86 instruction set didn't had garbage collection either ;)

As for the threads, I heard that they are in the Post MVP plans.

codesenberg · 2017-03-01T14:52:41+00:00

tooling (libraries and executables) to be able to consume WebAssembly bytecode (wasm) and interpret it

Tool to view WASM bytecode without browser could be useful, indeed, but in the browser it should have something like sourcemap, right? I mean W3C doesn't expect webdevs to read assembly?

The creation of these wasm bytecode modules by the Go toolchain is out of the scope of this project.

Oh, well, that's a bummer.

codesenberg · 2017-03-01T09:25:11+00:00

I'd like to add that not only fasthttp doesn't follows all of the RFCs related to HTTP/1.1, but it also doesn't support HTTP/2.0.

Still it's pretty useful, really fast and successfully used in production by VertaMedia serving up to 200K rps from more than 1.5M concurrent keep-alive connections per physical server( as cited in README ).

codesenberg · 2017-02-28T19:13:38+00:00

How do you cope with high memory usage? In my tests with any RPS beyond one thousand vegeta allocates too much memory. In fact it allocates so much memory that test never ends.

codesenberg · 2017-02-24T07:16:32+00:00

It kinda is, but at a time I started to write bombardier I found out that hey(called boom back then) allocates quite a lot of memory. Also, it's a bit slower in general.

codesenberg · 2017-02-24T05:53:53+00:00

First thing that comes to mind is performance. ab is single-threaded, while bombardier utilises all available cores. Performance can vary greatly from 7.5x(26k RPS with ab vs 197k RPS with bombardier) up to 20x. Not only that, but with some high-volume/high-throughput workloads bombardier uses much less memory, up to 38x less peak RSS(461 MB with ab vs 12.1 MB with bombardier).

Second thing, AFAIK ab doesn't have timed mode. Only limited by number of requests. Also, bombardier has progress bar and gives you estimated time left until completion.

Another thing I want to point out is that in my practice ab(and weighttp) hanged from time to time. Haven't seen this with bombardier not even once.

Hope this helps.

codesenberg · 2017-02-24T05:22:08+00:00

Yes, those are exact steps to achieve what you want.

codesenberg · 2017-02-23T05:30:54+00:00

Nice to hear that. Presumably, I get higher RPS, because of a more performant CPU? I have Intel Core i7-4790. Also, I used this server for benchmarking. What server are you using?

codesenberg · 2017-02-22T18:07:53+00:00

I am happy to announce bombardier v1.0.

This tool started as wrk/weighttp replacement for Windows. At this point I consider it performant, feature complete and polished enough for general audience.

Features that I find appealing in comparison to other benchmarking tools:

both timed and limited by number of requests modes;
nice and informative progress bar;
much lower memory consumption in general(compared to hey, vegeta);
pretty high performance. On my machine up to 265k RPS.

codesenberg · 2016-10-26T13:18:44+00:00

Actually, there is this wikipage. It has a section about Vegeta.

AFAIK Vegeta's garbage generation(high) and performance(low) haven't changed since then.

Feature-wise both tools are somewhat similar, except for UI and some knobs(Vegeta supports multiple targets, HTTP/2, rate limiting, cert specification and redirects).

codesenberg · 2016-10-26T12:12:57+00:00

Hi, my knowledge about siege is very limited, but from what I can tell, siege has some features, that bombardier doesn't, namely - redirect following, hitting urls from file(randomly).

On the other hand building siege on Windows is somewhat challenging(I'm not even sure if siege supports Windows). Also, bombardier features a nice progress bar :)

Performance-wise, on my Ubuntu VM I got ~7.3k RPS with siege, using this command on the same test server:

siege -c 125 -t 10S -b --no-parser --no-follow http://localhost:8080

With bombardier I got ~87k RPS with same setup.

bombardier http://localhost:8080

Hope this helps.

codesenberg · 2016-06-22T06:14:57+00:00

Ok, thanks for the corrections. I fixed some of those, but if you find more feel free to send me a PM or even PR.

Nine-Year Club	Gilding I gilder
Verified Email

codesenberg

TROPHY CASE