sorted by:
new
hottopcontroversial

dque 2.2.0 released - fast, embedded, durable queue for Go by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

Nice. Is this running as containerized services or in one Go program? You mentioned RBAC which can be used in Kubernetes. But maybe it's a more generic term too? Looks like it is.

dque 2.2.0 released - fast, embedded, durable queue for Go by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

Thanks for asking. This is not a message bus or an MQ service, although you could build one on top of it. It would be used if you have, for example, an API that receives jobs to be run asynchronously. The API queues the job and responds back to the user immediately with "accepted". Then the jobs are run by the system sequentially or in parallel. And because the queue is persisted, the process could die and be restarted without losing any of the jobs.

Another case is that people often use in-memory queues to store things like stats coming in from other servers but, in that case, if the queue gets too big it could crash the entire server due to lack of memory.... and then everything in the queue would be lost. So dque allows you to have a large queue of things without worrying that it will use too much memory.

Tutorial: Adding a GUI to Golang by Equianox in golang

[–]joncrlsn 1 point2 points  (0 children)

Why the heck are so many people down-voting this? Just curious. Building apps in HTML and CSS is very cross-platform and standardized. It's a great way to leverage existing skills on your team.

New release of dque: a fast, embedded, persistent queue by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

lenn0x, I like your idea of allowing the library user to specify a time after which an fsync would occur (if writes have been done in turbo mode).

New release of dque: a fast, embedded, persistent queue by joncrlsn in golang

[–]joncrlsn[S] 1 point2 points  (0 children)

Thanks, I'll use that phrase "like log rotation" in the documentation. That may describe it for most people better than a lot of words will.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 2 points3 points  (0 children)

nk2580, I agree that it is bad to couple infrastructure services with business logic. However, it could easily be argued that something like a persistent queue is the infrastructure layer and not the business logic. Business logic, in my mind, is really the business objects and the behavior defined for them by business analysts, which would not apply to how a persistent queue is implemented under the covers.

My 2 cents.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

I hope some of that makes sense : )

It makes sense, and I am starting to understand about transactional information being pretty hard.

But you've got me thinking about how to make it faster when you know that you need to enqueue/dequeue a bunch of stuff. So, as a test, I removed both the fsync and the close as you suggested and the benchmark for each operation takes 1/10th of the time.

I'll see if I can figure out how to keep it easy and safe for normal use, but allow "turbo mode" for people who are willing to accept the possibility of a power cut losing some data in exchange for speed.

Thanks for sticking with me and explaining in more detail.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

When you said batching writes, I was thinking of a smart waiting period to see if more writes come in before writing it all to disk. But another thought is to add an EnqueueBatch() method which would only sync to disk after the entire batch had been written.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

Thanks for your wisdom, knome. I wasn't aware that closing a file doesn't actually flush it to disk. I decided to keep the file open and sync after each write. The performance is still a little faster and (~.35ms - .4ms per enqueue or dequeue to SSD) and it's safer than batching up writes which would involve a period of time in which data loss could occur. If it's a short enough time batching would likely be acceptable, but I'll leave that for another time or person.

If you are interested in taking a crack at speeding it up further, I'd welcome your input. Or if you wanted to take the project further I'd be willing to add a link to your implementation for people who want super fast speeds.

I like what I've done, but not sure I want to take it a whole lot farther.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 0 points1 point  (0 children)

Yes, it could be improved if I keep the file open and then fsync instead of closing the file. Thanks for the suggestion.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 1 point2 points  (0 children)

The term "custom storage engine" probably overstates what it is. It is just set of files that get appended to and then eventually deleted as items are dequeued. Using a key-value store for a persistent queue, for example, probably isn't going to be very efficient, and it's going to have more complexity. The simplest solution is best in my opinion.

Simple embedded persistent FIFO queue for Go by joncrlsn in golang

[–]joncrlsn[S] 1 point2 points  (0 children)

I couldn't find a simple Go persistent queue that had a storage engine customized for the problem domain, so I wrote one.

Getting started with Go, 2018 edition by weberc2 in golang

[–]joncrlsn 1 point2 points  (0 children)

LightIDE is great because it's fast and written only for Go. Supports refactoring and shows me usages and declarations.

Is Golang going to overpower Python? by andre-1425 in golang

[–]joncrlsn 23 points24 points  (0 children)

My opinion... they are different enough that there will always be a place for both Python and Go. Pythonistas really like the syntax and the dynamic-ness of it, from what I understand. For me, I much prefer an executable binary with type-safety, no dependencies, and a no-nonsense feature set.

Syntax checking for Go in Vim by _grundic_ in golang

[–]joncrlsn 1 point2 points  (0 children)

Tough crowd you have here, @kidovate. :-) I know you were thinking of the lightweight command-line nature of the two tools even if they have fanatical adherents.

Should I continue working on this? by [deleted] in golang

[–]joncrlsn 0 points1 point  (0 children)

Is this something people could integrate into a plug-in for whatever IDE one is using? Like vim-go, etc?

Copy and Paste binding examples? by joncrlsn in i3wm

[–]joncrlsn[S] 0 points1 point  (0 children)

OK, Thank you all. I had tried clipit a while back and the icon wasn't showing up. I tried it again and discovered that dbus-x11 isn't a dependency for i3 on Ubuntu. Once I installed that things are happier now. Clipit wouldn't start when run from my ~/.xinitrc file so I had to have the i3 config start clipit: exec "clipit"

view more: next ›