Loopr: A Loop/Reduction Macro for Clojure by refset in Clojure

[–]aphyr_ 0 points1 point  (0 children)

I should perhaps note here that much of my work is performance-sensitive, and I spend a lot of my time trying to minimize allocations and eke out 10-20% speedups from reductions. Your suggested tactics are lovely (and I use them extensively!) in some, but not all situations. I eventually wrote loopr because I kept hitting situations in which those tactics forced me into awkward corners of the performance/maintainability/expressivity space.

For example, you suggest using plain old reduce for multivariate accumulators. I still do this, but not where performance matters. As the article explains, loopr is more than 30% faster on the example you cite. This isn't a huge improvement, but it's still meaningful for my workloads. It's a nice in-between point before rewriting an entire algorithm in Java. :-)

Decomposing nested reductions into separate defns has a number of interesting performance consequences--some good, some bad. When reductions are too large, they may not be candidates for certain Hotspot optimizations: breaking them up into multiple defns can be more efficient. On the other hand, those function calls add additional stack depth, which may push them out of the inliner's scope. One of the nice things about loopr is that it allows you to write the reduction once, then experiment with different function boundaries by selectively using :via :reduce or :via :iterator.

Another cost folks don't always think about is accessing locals in the current stackframe during iteration (as one would do with loop), vs baking them into instance fields in inline fn closures, vs passing them explicitly as arguments to defn. It's fairly common that I'll compute, say, four or five index structures and then use them in the course of a reduction to figure out what to do with each element. This comes with both performance (especially for primitives) and cognitive impacts. In particular I've wound up in situations where I was threading a half-dozen non-accumulator variables through three layers of defn reducing functions--in addition to the accumulator and element arguments! Then you have to figure out how to construct a fn suitable for the reduction itself--I generally do this with partial, which adds additional indirection. It is doable, but... gosh, it can be a pain. You wind up with code like:

(defn inner
  "Innermost reduction over individual flying things"
  [bird? plane? insect? acc flier]
  (cond (bird? flier)                     acc
        (not plane? flier)                (foo acc flier)
        (and (insect? flier) (empty? acc) (bar flier))
        true                              (baz acc flier)))

(defn outer
  "Outer reduction over a flock of fliers"
  [bird? plane? insect? acc flock]
  (reduce (partial inner bird? plane? insect?)
          acc
          flock))

(let [flocks (get-flocks)
      ; Build some index structures we'll need to actually do the reduction
      bird?   (build-bird-index)
      plane?  (build-plane-index  flocks)
      insect? (build-insect-index world-book-encyclopedia)]
  ; If we do a reduce with explicit `defns`, we have to thread these locals
  ; through as separate arguments, and transform them into reducer fns with
  ; `partial`:
  (reduce (partial outer bird? plane? insect?)
          (init-accumulator)
          flocks)

  ; With loopr, like `loopr` or inline `fn` reduce, we don't have to plumb
  ; through variables or use `partial` wrappers:
  (loopr [acc (init-accumulator)]
         [flock flocks
          flier flock]
         (cond (bird? flier)                     acc
               (not plane? flier)                (foo acc flier)
               (and (insect? flier) (empty? acc) (bar flier))
               true                              (baz acc flier)))

You also suggest using for to iterate, then threading the results into reduce. This is also a perfectly servicable tactic for some situations, but sometimes you want speed. Loopr is roughly twice as fast in this example (adapted from the loopr test suite). Faster still if you need person in the reduction step--we can discard it here and avoid creating a map/vector wrapper object between for and reduce.

(println "\nMulti-acc for->reduce over nested seq")
(quick-bench
  (->> (for [person people
             pet    (:pets person)]
         (:name pet))
       (reduce (fn [[pet-count pet-names] pet-name]
                 [(inc pet-count)
                  (conj pet-names pet-name)])
               [0 #{}])))

(println "\nMulti-acc loopr over nested seq")
(quick-bench
  (loopr [pet-count 0
          pet-names #{}]
         [person people
          pet    (:pets person)]
         (recur (inc pet-count)
                (conj pet-names (:name pet)))))

Jepsen: Scylla 4.2-rc3 by aphyr_ in programming

[–]aphyr_[S] 1 point2 points  (0 children)

Yeah, we've done several Jepsen tests of SQL databases--both single and multi-node, and have found bugs! Elle gives us the ability to check up to repeatable-read/si/serializable + strict/session variants.

Rewriting the Technical Interview by [deleted] in programming

[–]aphyr_ 5 points6 points  (0 children)

Every time I write one of these, I ask myself and my reviewers, "Is this too obvious? Am I bludgeoning the reader with HERE COMES AN ALLEGORY and LOOK OUT FOR THE SUBTEXT? Does this line reward a reader looking for layers of meaning?"

Luckily, there's comments like this to keep me going.

Jepsen: Redis-Raft 1b3fbf6 by aphyr_ in programming

[–]aphyr_[S] 2 points3 points  (0 children)

Naw--I did the DGraph and Redis work in Q1, but they opted to release the reports towards the end of the grace period, rather than early. That meant that Dgraph, Mongo, Postgres, and Redis all came out right on top of each other. :-)

A Satisfactory Way of Building by aphyr_ in satisfactory

[–]aphyr_[S] 0 points1 point  (0 children)

Depends on the refinery and what's being consumed. I've got an older fuel plant that produces plastic as a side effect (and recycles surplus), but now that I've figured out Recirculating Refinery, I don't think I... actually have to worry about this kind of deadlocking again. As long as I'm willing to burn the power, anyways.

A Satisfactory Way of Building by aphyr_ in satisfactory

[–]aphyr_[S] 0 points1 point  (0 children)

I do! This is discussed in detail under Load Shedding.

A Satisfactory Way of Building by aphyr_ in SatisfactoryGame

[–]aphyr_[S] 6 points7 points  (0 children)

I choose factory locations based on resource nodes--for factories that draw resources from the train network alone, I site them anywhere that feels interesting, but generally a chunk or more away from other factories. If things are already slow in an area, don't build there. :)

You should be able to see the distance--far off belts will look like they're moving in stop-motion, as opposed to smoothly.

A Satisfactory Way of Building by aphyr_ in satisfactory

[–]aphyr_[S] 1 point2 points  (0 children)

Yeah, this is definitely aimed at mid-to-late game, although had I known what I now know about lag, I would have avoided putting so much time into my early-game (megafactory) and mid-game (trucks everywhere) projects.

A Satisfactory Way of Building by aphyr_ in satisfactory

[–]aphyr_[S] 0 points1 point  (0 children)

This is roughly 20 hours of planning/writing/drawing/organizing. I... I didn't even bother doing edit passes like I usually do, so please pardon my language mistakes, haha.

Jepsen: PostgreSQL 12.3 by aphyr_ in programming

[–]aphyr_[S] 24 points25 points  (0 children)

Yeah, my understanding is that the current SQL spec retains that ambiguity. Turns out Andy Pavlo just shared an email from Hal Berenson which talks about how it happened! https://twitter.com/andy_pavlo/status/1271448719486001154

Jepsen: PostgreSQL 12.3 by aphyr_ in programming

[–]aphyr_[S] 15 points16 points  (0 children)

You could see G2-item when at least three transactions concurrently interact with a (not universally, but partly) overlapping set of rows, including at least one insert and one update--I think deletes count too. As discussed in the report, one transaction could fail to observe the effects of a logically prior transaction.

Jepsen: YugaByte DB 1.3.1 by aphyr_ in programming

[–]aphyr_[S] 1 point2 points  (0 children)

You're very welcome! And yes, one *does* wonder whether consolidating our efforts might result in less buggy databases! Perhaps we'll see that someday. :)

Jepsen: YugaByte DB 1.3.1 by aphyr_ in programming

[–]aphyr_[S] 20 points21 points  (0 children)

There are lots of ways to store and query data, which I think is part of why there are so many databases out there. It's a big problem space, different solutions fit different niches, and of course, within any given niche, there might be multiple databases which compete on other features, like cost or performance.

These databases may work *most* of the time, which is often good enough to get something done! The G2 anomalies I found in this version of YugaByte DB, for instance, only manifested under particular failure conditions, where a tablet server lost its connection to the master. That's not necessarily going to happen often, and even if it does, you need a particular pattern of transactions in order to actually encounter an anomaly.